[rank52]:[W1114 23:10:36.518601063 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 52] using GPU 4 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank13]:[W1114 23:11:12.238460370 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 13] using GPU 5 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank15]:[W1114 23:11:53.901505159 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 15] using GPU 7 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank34]:[W1114 23:12:26.422080415 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 34] using GPU 2 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank11]:[W1114 23:12:27.466992049 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 11] using GPU 3 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank35]:[W1114 23:12:34.418971192 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 35] using GPU 3 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank9]:[W1114 23:12:35.100500845 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 9] using GPU 1 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank10]:[W1114 23:12:40.584306199 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 10] using GPU 2 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank59]:[W1114 23:13:13.124187773 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 59] using GPU 3 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank26]:[W1114 23:13:14.401905249 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 26] using GPU 2 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank58]:[W1114 23:13:18.069590809 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 58] using GPU 2 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank56]:[W1114 23:13:24.803471906 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 56] using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank21]:[W1114 23:13:30.966391622 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 21] using GPU 5 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank16]:[W1114 23:13:30.305336727 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 16] using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank12]:[W1114 23:13:38.547967414 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 12] using GPU 4 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank42]:[W1114 23:13:38.415859500 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 42] using GPU 2 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank49]:[W1114 23:13:41.219292343 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 49] using GPU 1 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank57]:[W1114 23:13:42.378293248 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 57] using GPU 1 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank23]:[W1114 23:13:46.975292365 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 23] using GPU 7 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank37]:[W1114 23:13:51.850704805 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 37] using GPU 5 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank47]:[W1114 23:13:54.249835150 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 47] using GPU 7 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank22]:[W1114 23:13:58.469178190 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 22] using GPU 6 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank39]:[W1114 23:14:05.357492177 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 39] using GPU 7 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank38]:[W1114 23:14:07.947935687 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 38] using GPU 6 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank55]:[W1114 23:14:08.568878432 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 55] using GPU 7 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank14]:[W1114 23:14:11.686177516 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 14] using GPU 6 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank60]:[W1114 23:14:22.856066511 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 60] using GPU 4 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank63]:[W1114 23:14:25.960916721 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 63] using GPU 7 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank30]:[W1114 23:14:30.111806085 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 30] using GPU 6 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank46]:[W1114 23:14:31.764640785 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 46] using GPU 6 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank43]:[W1114 23:14:37.114226513 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 43] using GPU 3 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank27]:[W1114 23:14:42.389851235 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 27] using GPU 3 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank44]:[W1114 23:14:49.972119868 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 44] using GPU 4 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank61]:[W1114 23:15:02.407690030 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 61] using GPU 5 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank33]:[W1114 23:15:03.504110519 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 33] using GPU 1 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank36]:[W1114 23:15:08.828628810 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 36] using GPU 4 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank20]:[W1114 23:15:18.599089069 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 20] using GPU 4 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank51]:[W1114 23:15:19.029543082 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 51] using GPU 3 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank24]:[W1114 23:15:20.031216527 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 24] using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank18]:[W1114 23:15:25.016057988 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 18] using GPU 2 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank48]:[W1114 23:15:26.995062883 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 48] using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank62]:[W1114 23:15:29.116631378 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 62] using GPU 6 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank40]:[W1114 23:15:37.508155037 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 40] using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank29]:[W1114 23:15:37.427291528 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 29] using GPU 5 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank53]:[W1114 23:15:39.846950810 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 53] using GPU 5 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank17]:[W1114 23:15:49.450836512 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 17] using GPU 1 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank5]:[W1114 23:15:58.232626863 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 5] using GPU 5 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank8]:[W1114 23:15:58.698257106 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 8] using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank45]:[W1114 23:16:02.917999386 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 45] using GPU 5 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank50]:[W1114 23:16:05.370192818 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 50] using GPU 2 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank28]:[W1114 23:16:16.613835471 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 28] using GPU 4 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank19]:[W1114 23:16:19.127009235 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 19] using GPU 3 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank54]:[W1114 23:16:26.258877500 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 54] using GPU 6 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank25]:[W1114 23:16:35.048215940 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 25] using GPU 1 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank7]:[W1114 23:16:38.819115612 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 7] using GPU 7 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank31]:[W1114 23:17:28.416523693 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 31] using GPU 7 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank32]:[W1114 23:17:35.024277518 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 32] using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank41]:[W1114 23:17:39.504851392 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 41] using GPU 1 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank4]:[W1114 23:17:50.192734575 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 4] using GPU 4 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank0]:[W1114 23:18:14.408492884 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 0] using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank0]: Traceback (most recent call last): [rank0]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank0]: main() [rank0]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank0]: dist.barrier() [rank0]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank0]: return func(*args, **kwargs) [rank0]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank0]: work = group.barrier(opts=opts) [rank0]: torch.distributed.DistBackendError: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:2329, internal error - please report this issue to the NCCL developers, NCCL version 2.21.5 [rank0]: ncclInternalError: Internal check failed. [rank0]: Last error: [rank0]: Bootstrap : no socket interface found [rank0]:[W1114 23:18:15.102094023 ProcessGroupNCCL.cpp:1326] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator()) [rank4]: Traceback (most recent call last): [rank4]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank4]: main() [rank4]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank4]: dist.barrier() [rank4]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank4]: return func(*args, **kwargs) [rank4]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank4]: work = group.barrier(opts=opts) [rank4]: torch.distributed.DistBackendError: [4] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank4]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank4]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f88d8f6bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank4]: frame #1: + 0x6021e58 (0x7f88c8ae1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank4]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f88c8adebdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank4]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f88c8adef5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank4]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f88c8adfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank4]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f88c8a8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank4]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f88c8a8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank4]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f88c8a8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank4]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f887b75adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank4]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f887b7674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank4]: frame #10: + 0x1205255 (0x7f887b76d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank4]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f887b76e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank4]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f887b77f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank4]: frame #13: + 0x5fc28a2 (0x7f88c8a828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank4]: frame #14: + 0x5fcd035 (0x7f88c8a8d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank4]: frame #15: + 0x55d8ceb (0x7f88c8098ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank4]: frame #16: + 0x55d6579 (0x7f88c8096579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank4]: frame #17: + 0x1aa7828 (0x7f88c4567828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank4]: frame #18: + 0x5fd6d74 (0x7f88c8a96d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank4]: frame #19: + 0x5fd7b05 (0x7f88c8a97b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank4]: frame #20: + 0xe1df28 (0x7f88d8583f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank4]: frame #21: + 0x4cf144 (0x7f88d7c35144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank4]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank4]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank4]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank4]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank4]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank4]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank4]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank4]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank4]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank4]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank4]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank4]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank4]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank4]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank4]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank4]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank4]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank4]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank4]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank4]: frame #41: + 0x29d90 (0x7f88eefa9d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank4]: frame #42: __libc_start_main + 0x80 (0x7f88eefa9e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank4]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank4]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank5]: Traceback (most recent call last): [rank5]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank5]: main() [rank5]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank5]: dist.barrier() [rank5]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank5]: return func(*args, **kwargs) [rank5]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank5]: work = group.barrier(opts=opts) [rank5]: torch.distributed.DistBackendError: [5] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank5]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank5]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f1ecbd6bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank5]: frame #1: + 0x6021e58 (0x7f1ebb8e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank5]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f1ebb8debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank5]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f1ebb8def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank5]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f1ebb8dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank5]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f1ebb88ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank5]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f1ebb88ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank5]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f1ebb88ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank5]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f1e6e55adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank5]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f1e6e5674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank5]: frame #10: + 0x1205255 (0x7f1e6e56d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank5]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f1e6e56e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank5]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f1e6e57f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank5]: frame #13: + 0x5fc28a2 (0x7f1ebb8828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank5]: frame #14: + 0x5fcd035 (0x7f1ebb88d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank5]: frame #15: + 0x55d8ceb (0x7f1ebae98ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank7]: Traceback (most recent call last): [rank7]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank7]: main() [rank7]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank7]: dist.barrier() [rank7]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank7]: return func(*args, **kwargs) [rank7]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank7]: work = group.barrier(opts=opts) [rank7]: torch.distributed.DistBackendError: [7] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank7]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank7]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fb8548b8cf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank7]: frame #1: + 0x6021e58 (0x7fb8444e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank7]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7fb8444debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank7]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7fb8444def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank7]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7fb8444dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank7]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fb84448ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank7]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fb84448ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank7]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fb84448ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank7]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7fb7f715adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank7]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7fb7f71674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank7]: frame #10: + 0x1205255 (0x7fb7f716d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank7]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7fb7f716e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank7]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7fb7f717f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank7]: frame #13: + 0x5fc28a2 (0x7fb8444828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank7]: frame #14: + 0x5fcd035 (0x7fb84448d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank7]: frame #15: + 0x55d8ceb (0x7fb843a98ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank5]: frame #16: + 0x55d6579 (0x7f1ebae96579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank5]: frame #17: + 0x1aa7828 (0x7f1eb7367828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank5]: frame #18: + 0x5fd6d74 (0x7f1ebb896d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank5]: frame #19: + 0x5fd7b05 (0x7f1ebb897b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank5]: frame #20: + 0xe1df28 (0x7f1ecb383f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank5]: frame #21: + 0x4cf144 (0x7f1ecaa35144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank5]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank5]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank5]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank5]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank5]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank5]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank5]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank5]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank5]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank5]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank5]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank5]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank5]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank5]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank5]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank5]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank5]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank5]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank5]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank5]: frame #41: + 0x29d90 (0x7f1ee1d80d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank5]: frame #42: __libc_start_main + 0x80 (0x7f1ee1d80e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank5]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank5]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank7]: frame #16: + 0x55d6579 (0x7fb843a96579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank7]: frame #17: + 0x1aa7828 (0x7fb83ff67828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank7]: frame #18: + 0x5fd6d74 (0x7fb844496d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank7]: frame #19: + 0x5fd7b05 (0x7fb844497b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank7]: frame #20: + 0xe1df28 (0x7fb853f83f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank7]: frame #21: + 0x4cf144 (0x7fb853635144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank7]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank7]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank7]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank7]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank7]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank7]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank7]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank7]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank7]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank7]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank7]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank7]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank7]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank7]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank7]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank7]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank7]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank7]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank7]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank7]: frame #41: + 0x29d90 (0x7fb86a8fbd90 in /lib/x86_64-linux-gnu/libc.so.6) [rank7]: frame #42: __libc_start_main + 0x80 (0x7fb86a8fbe40 in /lib/x86_64-linux-gnu/libc.so.6) [rank7]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank7]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank16]: Traceback (most recent call last): [rank16]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank16]: main() [rank16]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank16]: dist.barrier() [rank16]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank16]: return func(*args, **kwargs) [rank16]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank16]: work = group.barrier(opts=opts) [rank16]: torch.distributed.DistBackendError: [16] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank16]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank16]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fc59f16bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank16]: frame #1: + 0x6021e58 (0x7fc58ece1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank16]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7fc58ecdebdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank16]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7fc58ecdef5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank16]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7fc58ecdfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank16]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc58ec8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank16]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc58ec8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank16]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc58ec8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank16]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7fc54195adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank16]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7fc5419674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank16]: frame #10: + 0x1205255 (0x7fc54196d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank16]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7fc54196e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank16]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7fc54197f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank16]: frame #13: + 0x5fc28a2 (0x7fc58ec828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank16]: frame #14: + 0x5fcd035 (0x7fc58ec8d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank16]: frame #15: + 0x55d8ceb (0x7fc58e298ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank26]: Traceback (most recent call last): [rank26]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank26]: main() [rank26]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank26]: dist.barrier() [rank26]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank26]: return func(*args, **kwargs) [rank26]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank26]: work = group.barrier(opts=opts) [rank26]: torch.distributed.DistBackendError: [26] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank26]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank16]: frame #16: + 0x55d6579 (0x7fc58e296579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank16]: frame #17: + 0x1aa7828 (0x7fc58a767828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank16]: frame #18: + 0x5fd6d74 (0x7fc58ec96d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank16]: frame #19: + 0x5fd7b05 (0x7fc58ec97b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank16]: frame #20: + 0xe1df28 (0x7fc59e783f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank16]: frame #21: + 0x4cf144 (0x7fc59de35144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank16]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank16]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank16]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank16]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank16]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank16]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank16]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank16]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank16]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank16]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank16]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank16]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank16]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank16]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank16]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank16]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank16]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank16]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank16]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank16]: frame #41: + 0x29d90 (0x7fc5b51cad90 in /lib/x86_64-linux-gnu/libc.so.6) [rank16]: frame #42: __libc_start_main + 0x80 (0x7fc5b51cae40 in /lib/x86_64-linux-gnu/libc.so.6) [rank16]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank16]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank18]: Traceback (most recent call last): [rank18]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank18]: main() [rank18]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank18]: dist.barrier() [rank18]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank18]: return func(*args, **kwargs) [rank18]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank18]: work = group.barrier(opts=opts) [rank18]: torch.distributed.DistBackendError: [18] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank18]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank18]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f35a0eb8cf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank18]: frame #1: + 0x6021e58 (0x7f3590ae1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank18]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f3590adebdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank18]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f3590adef5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank18]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f3590adfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank18]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f3590a8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank18]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f3590a8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank18]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f3590a8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank18]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f354375adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank18]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f35437674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank18]: frame #10: + 0x1205255 (0x7f354376d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank18]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f354376e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank18]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f354377f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank18]: frame #13: + 0x5fc28a2 (0x7f3590a828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank18]: frame #14: + 0x5fcd035 (0x7f3590a8d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank18]: frame #15: + 0x55d8ceb (0x7f3590098ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank18]: frame #16: + 0x55d6579 (0x7f3590096579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank18]: frame #17: + 0x1aa7828 (0x7f358c567828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank18]: frame #18: + 0x5fd6d74 (0x7f3590a96d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank18]: frame #19: + 0x5fd7b05 (0x7f3590a97b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank18]: frame #20: + 0xe1df28 (0x7f35a0583f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank18]: frame #21: + 0x4cf144 (0x7f359fc35144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank18]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank18]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank18]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank18]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank18]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank18]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank18]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank18]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank18]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank18]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank18]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank18]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank18]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank18]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank18]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank18]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank18]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank18]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank18]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank18]: frame #41: + 0x29d90 (0x7f35b6e67d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank18]: frame #42: __libc_start_main + 0x80 (0x7f35b6e67e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank18]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank18]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank26]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fafcaf6bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank26]: frame #1: + 0x6021e58 (0x7fafbaae1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank26]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7fafbaadebdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank26]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7fafbaadef5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank26]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7fafbaadfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank21]: Traceback (most recent call last): [rank21]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank21]: main() [rank21]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank21]: dist.barrier() [rank21]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank21]: return func(*args, **kwargs) [rank21]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank21]: work = group.barrier(opts=opts) [rank21]: torch.distributed.DistBackendError: [21] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank21]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank21]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f85786b8cf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank21]: frame #1: + 0x6021e58 (0x7f85682e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank21]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f85682debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank21]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f85682def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank21]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f85682dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank21]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f856828ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank21]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f856828ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank21]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f856828ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank21]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f851af5adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank21]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f851af674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank21]: frame #10: + 0x1205255 (0x7f851af6d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank21]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f851af6e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank21]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f851af7f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank21]: frame #13: + 0x5fc28a2 (0x7f85682828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank21]: frame #14: + 0x5fcd035 (0x7f856828d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank21]: frame #15: + 0x55d8ceb (0x7f8567898ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank17]: Traceback (most recent call last): [rank17]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank17]: main() [rank17]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank17]: dist.barrier() [rank17]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank17]: return func(*args, **kwargs) [rank17]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank17]: work = group.barrier(opts=opts) [rank17]: torch.distributed.DistBackendError: [17] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank17]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank17]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f32e70d3cf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank17]: frame #1: + 0x6021e58 (0x7f33356e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank17]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f33356debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank17]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f33356def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank17]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f33356dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank17]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f333568ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank17]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f333568ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank17]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f333568ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank17]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f32e835adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank17]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f32e83674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank17]: frame #10: + 0x1205255 (0x7f32e836d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank17]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f32e836e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank17]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f32e837f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank17]: frame #13: + 0x5fc28a2 (0x7f33356828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank12]: Traceback (most recent call last): [rank12]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank12]: main() [rank12]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank12]: dist.barrier() [rank12]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank12]: return func(*args, **kwargs) [rank12]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank12]: work = group.barrier(opts=opts) [rank12]: torch.distributed.DistBackendError: [12] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank12]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank17]: frame #14: + 0x5fcd035 (0x7f333568d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank17]: frame #15: + 0x55d8ceb (0x7f3334c98ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank21]: frame #16: + 0x55d6579 (0x7f8567896579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank21]: frame #17: + 0x1aa7828 (0x7f8563d67828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank21]: frame #18: + 0x5fd6d74 (0x7f8568296d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank21]: frame #19: + 0x5fd7b05 (0x7f8568297b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank21]: frame #20: + 0xe1df28 (0x7f8577d83f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank21]: frame #21: + 0x4cf144 (0x7f8577435144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank21]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank21]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank21]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank21]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank21]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank21]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank21]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank21]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank21]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank21]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank21]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank21]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank21]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank21]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank21]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank21]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank21]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank21]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank21]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank21]: frame #41: + 0x29d90 (0x7f858e6d1d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank21]: frame #42: __libc_start_main + 0x80 (0x7f858e6d1e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank21]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank21]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank17]: frame #16: + 0x55d6579 (0x7f3334c96579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank17]: frame #17: + 0x1aa7828 (0x7f3331167828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank17]: frame #18: + 0x5fd6d74 (0x7f3335696d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank17]: frame #19: + 0x5fd7b05 (0x7f3335697b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank17]: frame #20: + 0xe1df28 (0x7f3345183f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank17]: frame #21: + 0x4cf144 (0x7f3344835144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank17]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank17]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank17]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank17]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank17]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank17]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank17]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank17]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank17]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank17]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank17]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank17]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank17]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank17]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank17]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank17]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank17]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank17]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank17]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank17]: frame #41: + 0x29d90 (0x7f335ba28d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank17]: frame #42: __libc_start_main + 0x80 (0x7f335ba28e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank17]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank17]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank19]: Traceback (most recent call last): [rank19]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank19]: main() [rank19]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank19]: dist.barrier() [rank19]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank19]: return func(*args, **kwargs) [rank19]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank19]: work = group.barrier(opts=opts) [rank19]: torch.distributed.DistBackendError: [19] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank19]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank19]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f830cd6bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank19]: frame #1: + 0x6021e58 (0x7f82fc8e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank19]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f82fc8debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank19]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f82fc8def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank19]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f82fc8dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank19]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f82fc88ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank19]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f82fc88ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank19]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f82fc88ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank19]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f82af55adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank19]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f82af5674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank19]: frame #10: + 0x1205255 (0x7f82af56d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank19]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f82af56e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank19]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f82af57f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank19]: frame #13: + 0x5fc28a2 (0x7f82fc8828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank19]: frame #14: + 0x5fcd035 (0x7f82fc88d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank19]: frame #15: + 0x55d8ceb (0x7f82fbe98ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank20]: Traceback (most recent call last): [rank20]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank20]: main() [rank20]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank20]: dist.barrier() [rank20]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank20]: return func(*args, **kwargs) [rank20]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank20]: work = group.barrier(opts=opts) [rank20]: torch.distributed.DistBackendError: [20] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank20]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank20]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fb8992b8cf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank20]: frame #1: + 0x6021e58 (0x7fb888ee1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank20]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7fb888edebdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank20]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7fb888edef5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank20]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7fb888edfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank26]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fafbaa8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank26]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fafbaa8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank26]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fafbaa8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank26]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7faf6d75adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank20]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fb888e8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank20]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fb888e8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank20]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fb888e8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank20]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7fb83bb5adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank20]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7fb83bb674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank20]: frame #10: + 0x1205255 (0x7fb83bb6d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank20]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7fb83bb6e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank20]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7fb83bb7f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank20]: frame #13: + 0x5fc28a2 (0x7fb888e828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank20]: frame #14: + 0x5fcd035 (0x7fb888e8d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank20]: frame #15: + 0x55d8ceb (0x7fb888498ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank22]: Traceback (most recent call last): [rank22]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank22]: main() [rank22]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank22]: dist.barrier() [rank22]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank22]: return func(*args, **kwargs) [rank22]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank22]: work = group.barrier(opts=opts) [rank22]: torch.distributed.DistBackendError: [22] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank22]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank22]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fef4136bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank22]: frame #1: + 0x6021e58 (0x7fef30ee1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank22]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7fef30edebdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank22]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7fef30edef5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank22]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7fef30edfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank22]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fef30e8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank22]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fef30e8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank22]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fef30e8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank22]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7feee3b5adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank26]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7faf6d7674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank26]: frame #10: + 0x1205255 (0x7faf6d76d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank26]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7faf6d76e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank26]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7faf6d77f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank26]: frame #13: + 0x5fc28a2 (0x7fafbaa828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank22]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7feee3b674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank22]: frame #10: + 0x1205255 (0x7feee3b6d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank22]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7feee3b6e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank22]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7feee3b7f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank22]: frame #13: + 0x5fc28a2 (0x7fef30e828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank22]: frame #14: + 0x5fcd035 (0x7fef30e8d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank22]: frame #15: + 0x55d8ceb (0x7fef30498ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank23]: Traceback (most recent call last): [rank23]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank23]: main() [rank23]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank23]: dist.barrier() [rank23]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank23]: return func(*args, **kwargs) [rank23]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank23]: work = group.barrier(opts=opts) [rank23]: torch.distributed.DistBackendError: [23] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank23]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank23]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f4dfcf6bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank23]: frame #1: + 0x6021e58 (0x7f4decae1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank23]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f4decadebdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank23]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f4decadef5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank23]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f4decadfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank26]: frame #14: + 0x5fcd035 (0x7fafbaa8d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank26]: frame #15: + 0x55d8ceb (0x7fafba098ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank23]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f4deca8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank23]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f4deca8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank23]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f4deca8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank23]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f4d9f75adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank25]: Traceback (most recent call last): [rank25]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank25]: main() [rank25]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank25]: dist.barrier() [rank25]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank25]: return func(*args, **kwargs) [rank25]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank25]: work = group.barrier(opts=opts) [rank25]: torch.distributed.DistBackendError: [25] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank25]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank23]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f4d9f7674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank23]: frame #10: + 0x1205255 (0x7f4d9f76d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank23]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f4d9f76e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank23]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f4d9f77f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank23]: frame #13: + 0x5fc28a2 (0x7f4deca828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank25]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f1aec2d3cf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank25]: frame #1: + 0x6021e58 (0x7f1b3a8e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank25]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f1b3a8debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank25]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f1b3a8def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank25]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f1b3a8dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank23]: frame #14: + 0x5fcd035 (0x7f4deca8d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank23]: frame #15: + 0x55d8ceb (0x7f4dec098ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank25]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f1b3a88ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank25]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f1b3a88ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank25]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f1b3a88ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank25]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f1aed55adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank19]: frame #16: + 0x55d6579 (0x7f82fbe96579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank19]: frame #17: + 0x1aa7828 (0x7f82f8367828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank19]: frame #18: + 0x5fd6d74 (0x7f82fc896d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank19]: frame #19: + 0x5fd7b05 (0x7f82fc897b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank19]: frame #20: + 0xe1df28 (0x7f830c383f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank19]: frame #21: + 0x4cf144 (0x7f830ba35144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank25]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f1aed5674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank25]: frame #10: + 0x1205255 (0x7f1aed56d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank25]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f1aed56e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank25]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f1aed57f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank25]: frame #13: + 0x5fc28a2 (0x7f1b3a8828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank19]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank19]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank19]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank19]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank19]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank19]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank19]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank19]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank19]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank25]: frame #14: + 0x5fcd035 (0x7f1b3a88d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank25]: frame #15: + 0x55d8ceb (0x7f1b39e98ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank19]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank19]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank19]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank19]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank19]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank19]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank19]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank19]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank19]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank19]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank26]: frame #16: + 0x55d6579 (0x7fafba096579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank26]: frame #17: + 0x1aa7828 (0x7fafb6567828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank26]: frame #18: + 0x5fd6d74 (0x7fafbaa96d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank26]: frame #19: + 0x5fd7b05 (0x7fafbaa97b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank26]: frame #20: + 0xe1df28 (0x7fafca583f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank26]: frame #21: + 0x4cf144 (0x7fafc9c35144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank19]: frame #41: + 0x29d90 (0x7f8322d2dd90 in /lib/x86_64-linux-gnu/libc.so.6) [rank19]: frame #42: __libc_start_main + 0x80 (0x7f8322d2de40 in /lib/x86_64-linux-gnu/libc.so.6) [rank19]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank19]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank26]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank26]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank26]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank26]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank26]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank26]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank26]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank26]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank26]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank20]: frame #16: + 0x55d6579 (0x7fb888496579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank20]: frame #17: + 0x1aa7828 (0x7fb884967828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank20]: frame #18: + 0x5fd6d74 (0x7fb888e96d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank20]: frame #19: + 0x5fd7b05 (0x7fb888e97b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank20]: frame #20: + 0xe1df28 (0x7fb898983f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank20]: frame #21: + 0x4cf144 (0x7fb898035144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank26]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank26]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank26]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank26]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank26]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank26]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank26]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank26]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank26]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank26]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank20]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank20]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank20]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank20]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank20]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank20]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank20]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank20]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank20]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank26]: frame #41: + 0x29d90 (0x7fafe0fb1d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank26]: frame #42: __libc_start_main + 0x80 (0x7fafe0fb1e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank26]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank26]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank20]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank20]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank20]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank20]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank20]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank20]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank20]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank20]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank20]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank20]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank31]: Traceback (most recent call last): [rank31]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank31]: main() [rank31]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank31]: dist.barrier() [rank31]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank31]: return func(*args, **kwargs) [rank31]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank31]: work = group.barrier(opts=opts) [rank31]: torch.distributed.DistBackendError: [31] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank31]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank20]: frame #41: + 0x29d90 (0x7fb8af276d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank20]: frame #42: __libc_start_main + 0x80 (0x7fb8af276e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank20]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank20]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank31]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fdaeb4b8cf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank31]: frame #1: + 0x6021e58 (0x7fdadb0e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank31]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7fdadb0debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank31]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7fdadb0def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank31]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7fdadb0dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank22]: frame #16: + 0x55d6579 (0x7fef30496579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank22]: frame #17: + 0x1aa7828 (0x7fef2c967828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank22]: frame #18: + 0x5fd6d74 (0x7fef30e96d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank22]: frame #19: + 0x5fd7b05 (0x7fef30e97b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank22]: frame #20: + 0xe1df28 (0x7fef40983f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank22]: frame #21: + 0x4cf144 (0x7fef40035144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank12]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f0ea256bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank12]: frame #1: + 0x6021e58 (0x7f0e920e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank12]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f0e920debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank12]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f0e920def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank12]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f0e920dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank31]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fdadb08ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank31]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fdadb08ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank31]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fdadb08ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank31]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7fda8dd5adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank22]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank22]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank22]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank22]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank22]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank22]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank22]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank22]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank22]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank31]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7fda8dd674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank31]: frame #10: + 0x1205255 (0x7fda8dd6d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank31]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7fda8dd6e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank31]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7fda8dd7f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank31]: frame #13: + 0x5fc28a2 (0x7fdadb0828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank22]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank22]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank22]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank22]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank22]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank22]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank22]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank22]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank22]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank22]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank31]: frame #14: + 0x5fcd035 (0x7fdadb08d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank31]: frame #15: + 0x55d8ceb (0x7fdada698ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank22]: frame #41: + 0x29d90 (0x7fef573b9d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank22]: frame #42: __libc_start_main + 0x80 (0x7fef573b9e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank22]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank22]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank24]: Traceback (most recent call last): [rank24]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank24]: main() [rank24]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank24]: dist.barrier() [rank24]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank24]: return func(*args, **kwargs) [rank24]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank24]: work = group.barrier(opts=opts) [rank24]: torch.distributed.DistBackendError: [24] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank24]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank23]: frame #16: + 0x55d6579 (0x7f4dec096579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank23]: frame #17: + 0x1aa7828 (0x7f4de8567828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank23]: frame #18: + 0x5fd6d74 (0x7f4deca96d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank23]: frame #19: + 0x5fd7b05 (0x7f4deca97b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank23]: frame #20: + 0xe1df28 (0x7f4dfc583f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank23]: frame #21: + 0x4cf144 (0x7f4dfbc35144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank24]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fe218f6bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank24]: frame #1: + 0x6021e58 (0x7fe208ae1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank24]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7fe208adebdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank24]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7fe208adef5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank24]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7fe208adfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank23]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank23]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank23]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank23]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank23]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank23]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank23]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank23]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank23]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank24]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fe208a8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank24]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fe208a8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank24]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fe208a8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank24]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7fe1bb75adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank23]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank23]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank23]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank23]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank23]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank23]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank23]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank23]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank23]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank23]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank24]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7fe1bb7674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank24]: frame #10: + 0x1205255 (0x7fe1bb76d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank24]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7fe1bb76e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank24]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7fe1bb77f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank24]: frame #13: + 0x5fc28a2 (0x7fe208a828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank23]: frame #41: + 0x29d90 (0x7f4e12fe8d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank23]: frame #42: __libc_start_main + 0x80 (0x7f4e12fe8e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank23]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank23]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank24]: frame #14: + 0x5fcd035 (0x7fe208a8d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank24]: frame #15: + 0x55d8ceb (0x7fe208098ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank12]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f0e9208ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank12]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f0e9208ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank12]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f0e9208ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank12]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f0e44d5adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank25]: frame #16: + 0x55d6579 (0x7f1b39e96579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank25]: frame #17: + 0x1aa7828 (0x7f1b36367828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank25]: frame #18: + 0x5fd6d74 (0x7f1b3a896d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank25]: frame #19: + 0x5fd7b05 (0x7f1b3a897b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank25]: frame #20: + 0xe1df28 (0x7f1b4a383f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank25]: frame #21: + 0x4cf144 (0x7f1b49a35144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank12]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f0e44d674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank12]: frame #10: + 0x1205255 (0x7f0e44d6d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank12]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f0e44d6e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank12]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f0e44d7f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank12]: frame #13: + 0x5fc28a2 (0x7f0e920828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank25]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank25]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank25]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank25]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank25]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank25]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank25]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank25]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank25]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank12]: frame #14: + 0x5fcd035 (0x7f0e9208d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank12]: frame #15: + 0x55d8ceb (0x7f0e91698ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank25]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank25]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank25]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank25]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank25]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank25]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank25]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank25]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank25]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank25]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank8]: Traceback (most recent call last): [rank8]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank8]: main() [rank8]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank8]: dist.barrier() [rank8]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank8]: return func(*args, **kwargs) [rank8]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank8]: work = group.barrier(opts=opts) [rank8]: torch.distributed.DistBackendError: [8] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank8]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank25]: frame #41: + 0x29d90 (0x7f1b60c41d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank25]: frame #42: __libc_start_main + 0x80 (0x7f1b60c41e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank25]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank25]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank8]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f192c2d3cf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank8]: frame #1: + 0x6021e58 (0x7f197a8e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank8]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f197a8debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank8]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f197a8def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank8]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f197a8dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank28]: Traceback (most recent call last): [rank28]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank28]: main() [rank28]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank28]: dist.barrier() [rank28]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank28]: return func(*args, **kwargs) [rank28]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank28]: work = group.barrier(opts=opts) [rank28]: torch.distributed.DistBackendError: [28] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank28]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank8]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f197a88ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank8]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f197a88ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank8]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f197a88ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank8]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f192d55adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank8]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f192d5674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank28]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f357736bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank28]: frame #1: + 0x6021e58 (0x7f3566ee1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank28]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f3566edebdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank28]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f3566edef5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank28]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f3566edfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank8]: frame #10: + 0x1205255 (0x7f192d56d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank8]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f192d56e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank8]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f192d57f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank8]: frame #13: + 0x5fc28a2 (0x7f197a8828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank8]: frame #14: + 0x5fcd035 (0x7f197a88d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank28]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f3566e8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank28]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f3566e8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank28]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f3566e8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank28]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f3519b5adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank8]: frame #15: + 0x55d8ceb (0x7f1979e98ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank28]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f3519b674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank28]: frame #10: + 0x1205255 (0x7f3519b6d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank28]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f3519b6e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank28]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f3519b7f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank28]: frame #13: + 0x5fc28a2 (0x7f3566e828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank9]: Traceback (most recent call last): [rank9]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank9]: main() [rank9]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank9]: dist.barrier() [rank9]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank9]: return func(*args, **kwargs) [rank9]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank9]: work = group.barrier(opts=opts) [rank9]: torch.distributed.DistBackendError: [9] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank9]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank28]: frame #14: + 0x5fcd035 (0x7f3566e8d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank28]: frame #15: + 0x55d8ceb (0x7f3566498ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank9]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f1f9296bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank9]: frame #1: + 0x6021e58 (0x7f1f824e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank9]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f1f824debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank9]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f1f824def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank9]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f1f824dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank29]: Traceback (most recent call last): [rank29]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank29]: main() [rank29]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank29]: dist.barrier() [rank29]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank29]: return func(*args, **kwargs) [rank29]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank29]: work = group.barrier(opts=opts) [rank29]: torch.distributed.DistBackendError: [29] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank29]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank9]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f1f8248ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank9]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f1f8248ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank9]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f1f8248ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank9]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f1f3515adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank9]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f1f351674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank29]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7ff87dd6bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank29]: frame #1: + 0x6021e58 (0x7ff86d8e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank29]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7ff86d8debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank29]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7ff86d8def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank29]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7ff86d8dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank9]: frame #10: + 0x1205255 (0x7f1f3516d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank9]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f1f3516e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank9]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f1f3517f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank9]: frame #13: + 0x5fc28a2 (0x7f1f824828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank9]: frame #14: + 0x5fcd035 (0x7f1f8248d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank29]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7ff86d88ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank29]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7ff86d88ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank29]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7ff86d88ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank29]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7ff82055adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank9]: frame #15: + 0x55d8ceb (0x7f1f81a98ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank29]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7ff8205674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank29]: frame #10: + 0x1205255 (0x7ff82056d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank29]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7ff82056e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank29]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7ff82057f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank29]: frame #13: + 0x5fc28a2 (0x7ff86d8828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank11]: Traceback (most recent call last): [rank11]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank11]: main() [rank11]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank11]: dist.barrier() [rank11]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank11]: return func(*args, **kwargs) [rank11]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank11]: work = group.barrier(opts=opts) [rank11]: torch.distributed.DistBackendError: [11] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank11]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank29]: frame #14: + 0x5fcd035 (0x7ff86d88d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank29]: frame #15: + 0x55d8ceb (0x7ff86ce98ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank11]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f1a6e16bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank11]: frame #1: + 0x6021e58 (0x7f1a5dce1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank11]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f1a5dcdebdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank11]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f1a5dcdef5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank11]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f1a5dcdfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank30]: Traceback (most recent call last): [rank30]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank30]: main() [rank30]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank30]: dist.barrier() [rank30]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank30]: return func(*args, **kwargs) [rank30]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank30]: work = group.barrier(opts=opts) [rank30]: torch.distributed.DistBackendError: [30] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank30]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank11]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f1a5dc8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank11]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f1a5dc8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank11]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f1a5dc8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank11]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f1a1095adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank30]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fc482eb8cf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank30]: frame #1: + 0x6021e58 (0x7fc472ae1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank30]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7fc472adebdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank30]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7fc472adef5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank30]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7fc472adfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank11]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f1a109674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank11]: frame #10: + 0x1205255 (0x7f1a1096d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank11]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f1a1096e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank11]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f1a1097f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank11]: frame #13: + 0x5fc28a2 (0x7f1a5dc828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank30]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc472a8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank30]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc472a8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank30]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc472a8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank30]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7fc42575adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank11]: frame #14: + 0x5fcd035 (0x7f1a5dc8d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank11]: frame #15: + 0x55d8ceb (0x7f1a5d298ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank30]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7fc4257674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank30]: frame #10: + 0x1205255 (0x7fc42576d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank30]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7fc42576e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank30]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7fc42577f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank30]: frame #13: + 0x5fc28a2 (0x7fc472a828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank12]: frame #16: + 0x55d6579 (0x7f0e91696579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank12]: frame #17: + 0x1aa7828 (0x7f0e8db67828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank12]: frame #18: + 0x5fd6d74 (0x7f0e92096d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank12]: frame #19: + 0x5fd7b05 (0x7f0e92097b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank12]: frame #20: + 0xe1df28 (0x7f0ea1b83f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank12]: frame #21: + 0x4cf144 (0x7f0ea1235144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank30]: frame #14: + 0x5fcd035 (0x7fc472a8d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank30]: frame #15: + 0x55d8ceb (0x7fc472098ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank12]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank12]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank12]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank12]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank12]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank12]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank12]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank12]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank12]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank31]: frame #16: + 0x55d6579 (0x7fdada696579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank31]: frame #17: + 0x1aa7828 (0x7fdad6b67828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank31]: frame #18: + 0x5fd6d74 (0x7fdadb096d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank31]: frame #19: + 0x5fd7b05 (0x7fdadb097b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank31]: frame #20: + 0xe1df28 (0x7fdaeab83f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank31]: frame #21: + 0x4cf144 (0x7fdaea235144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank12]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank12]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank12]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank12]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank12]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank12]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank12]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank12]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank12]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank12]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank31]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank31]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank31]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank31]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank31]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank31]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank31]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank31]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank31]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank12]: frame #41: + 0x29d90 (0x7f0eb85ffd90 in /lib/x86_64-linux-gnu/libc.so.6) [rank12]: frame #42: __libc_start_main + 0x80 (0x7f0eb85ffe40 in /lib/x86_64-linux-gnu/libc.so.6) [rank12]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank12]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank31]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank31]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank31]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank31]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank31]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank31]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank31]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank31]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank31]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank31]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank14]: Traceback (most recent call last): [rank14]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank14]: main() [rank14]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank14]: dist.barrier() [rank14]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank14]: return func(*args, **kwargs) [rank14]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank14]: work = group.barrier(opts=opts) [rank14]: torch.distributed.DistBackendError: [14] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank14]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank31]: frame #41: + 0x29d90 (0x7fdb01500d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank31]: frame #42: __libc_start_main + 0x80 (0x7fdb01500e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank31]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank31]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank14]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7efdc0b6bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank14]: frame #1: + 0x6021e58 (0x7efdb06e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank14]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7efdb06debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank14]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7efdb06def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank14]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7efdb06dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank24]: frame #16: + 0x55d6579 (0x7fe208096579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank24]: frame #17: + 0x1aa7828 (0x7fe204567828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank24]: frame #18: + 0x5fd6d74 (0x7fe208a96d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank24]: frame #19: + 0x5fd7b05 (0x7fe208a97b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank24]: frame #20: + 0xe1df28 (0x7fe218583f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank24]: frame #21: + 0x4cf144 (0x7fe217c35144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank14]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7efdb068ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank14]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7efdb068ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank14]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7efdb068ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank14]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7efd6335adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank24]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank24]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank24]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank24]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank24]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank24]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank24]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank24]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank24]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank14]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7efd633674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank14]: frame #10: + 0x1205255 (0x7efd6336d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank14]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7efd6336e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank14]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7efd6337f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank14]: frame #13: + 0x5fc28a2 (0x7efdb06828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank24]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank24]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank24]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank24]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank24]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank24]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank24]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank24]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank24]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank24]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank14]: frame #14: + 0x5fcd035 (0x7efdb068d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank14]: frame #15: + 0x55d8ceb (0x7efdafc98ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank24]: frame #41: + 0x29d90 (0x7fe22efa7d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank24]: frame #42: __libc_start_main + 0x80 (0x7fe22efa7e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank24]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank24]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank15]: Traceback (most recent call last): [rank15]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank15]: main() [rank15]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank15]: dist.barrier() [rank15]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank15]: return func(*args, **kwargs) [rank15]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank15]: work = group.barrier(opts=opts) [rank15]: torch.distributed.DistBackendError: [15] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank15]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank27]: Traceback (most recent call last): [rank27]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank27]: main() [rank27]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank27]: dist.barrier() [rank27]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank27]: return func(*args, **kwargs) [rank27]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank27]: work = group.barrier(opts=opts) [rank27]: torch.distributed.DistBackendError: [27] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank27]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank15]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f467356bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank15]: frame #1: + 0x6021e58 (0x7f46630e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank15]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f46630debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank15]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f46630def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank15]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f46630dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank27]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fbee416bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank27]: frame #1: + 0x6021e58 (0x7fbed3ce1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank27]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7fbed3cdebdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank27]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7fbed3cdef5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank27]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7fbed3cdfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank15]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f466308ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank15]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f466308ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank15]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f466308ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank15]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f4615d5adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank27]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fbed3c8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank27]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fbed3c8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank27]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fbed3c8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank27]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7fbe8695adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank15]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f4615d674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank15]: frame #10: + 0x1205255 (0x7f4615d6d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank15]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f4615d6e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank15]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f4615d7f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank15]: frame #13: + 0x5fc28a2 (0x7f46630828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank27]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7fbe869674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank27]: frame #10: + 0x1205255 (0x7fbe8696d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank27]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7fbe8696e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank27]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7fbe8697f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank27]: frame #13: + 0x5fc28a2 (0x7fbed3c828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank15]: frame #14: + 0x5fcd035 (0x7f466308d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank15]: frame #15: + 0x55d8ceb (0x7f4662698ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank27]: frame #14: + 0x5fcd035 (0x7fbed3c8d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank27]: frame #15: + 0x55d8ceb (0x7fbed3298ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank8]: frame #16: + 0x55d6579 (0x7f1979e96579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank8]: frame #17: + 0x1aa7828 (0x7f1976367828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank8]: frame #18: + 0x5fd6d74 (0x7f197a896d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank8]: frame #19: + 0x5fd7b05 (0x7f197a897b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank8]: frame #20: + 0xe1df28 (0x7f198a383f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank8]: frame #21: + 0x4cf144 (0x7f1989a35144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank28]: frame #16: + 0x55d6579 (0x7f3566496579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank28]: frame #17: + 0x1aa7828 (0x7f3562967828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank28]: frame #18: + 0x5fd6d74 (0x7f3566e96d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank28]: frame #19: + 0x5fd7b05 (0x7f3566e97b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank28]: frame #20: + 0xe1df28 (0x7f3576983f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank28]: frame #21: + 0x4cf144 (0x7f3576035144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank8]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank8]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank8]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank8]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank8]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank8]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank8]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank8]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank8]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank28]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank28]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank28]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank28]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank28]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank28]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank28]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank28]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank28]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank8]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank8]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank8]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank8]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank8]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank8]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank8]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank8]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank8]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank8]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank28]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank28]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank28]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank28]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank28]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank28]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank28]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank28]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank28]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank28]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank8]: frame #41: + 0x29d90 (0x7f19a0c57d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank8]: frame #42: __libc_start_main + 0x80 (0x7f19a0c57e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank8]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank8]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank28]: frame #41: + 0x29d90 (0x7f358d3aad90 in /lib/x86_64-linux-gnu/libc.so.6) [rank28]: frame #42: __libc_start_main + 0x80 (0x7f358d3aae40 in /lib/x86_64-linux-gnu/libc.so.6) [rank28]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank28]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank9]: frame #16: + 0x55d6579 (0x7f1f81a96579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank9]: frame #17: + 0x1aa7828 (0x7f1f7df67828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank9]: frame #18: + 0x5fd6d74 (0x7f1f82496d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank9]: frame #19: + 0x5fd7b05 (0x7f1f82497b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank9]: frame #20: + 0xe1df28 (0x7f1f91f83f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank9]: frame #21: + 0x4cf144 (0x7f1f91635144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank29]: frame #16: + 0x55d6579 (0x7ff86ce96579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank29]: frame #17: + 0x1aa7828 (0x7ff869367828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank29]: frame #18: + 0x5fd6d74 (0x7ff86d896d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank29]: frame #19: + 0x5fd7b05 (0x7ff86d897b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank29]: frame #20: + 0xe1df28 (0x7ff87d383f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank29]: frame #21: + 0x4cf144 (0x7ff87ca35144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank9]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank9]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank9]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank9]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank9]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank9]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank9]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank9]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank9]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank29]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank29]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank29]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank29]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank29]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank29]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank29]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank29]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank29]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank9]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank9]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank9]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank9]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank9]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank9]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank9]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank9]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank9]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank9]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank29]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank29]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank29]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank29]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank29]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank29]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank29]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank29]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank29]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank29]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank9]: frame #41: + 0x29d90 (0x7f1fa89a9d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank9]: frame #42: __libc_start_main + 0x80 (0x7f1fa89a9e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank9]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank9]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank29]: frame #41: + 0x29d90 (0x7ff893d47d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank29]: frame #42: __libc_start_main + 0x80 (0x7ff893d47e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank29]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank29]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank10]: Traceback (most recent call last): [rank10]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank10]: main() [rank10]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank10]: dist.barrier() [rank10]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank10]: return func(*args, **kwargs) [rank10]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank10]: work = group.barrier(opts=opts) [rank10]: torch.distributed.DistBackendError: [10] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank10]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank30]: frame #16: + 0x55d6579 (0x7fc472096579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank30]: frame #17: + 0x1aa7828 (0x7fc46e567828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank30]: frame #18: + 0x5fd6d74 (0x7fc472a96d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank30]: frame #19: + 0x5fd7b05 (0x7fc472a97b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank30]: frame #20: + 0xe1df28 (0x7fc482583f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank30]: frame #21: + 0x4cf144 (0x7fc481c35144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank10]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fc97576bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank10]: frame #1: + 0x6021e58 (0x7fc9652e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank10]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7fc9652debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank10]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7fc9652def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank10]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7fc9652dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank30]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank30]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank30]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank30]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank30]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank30]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank30]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank30]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank30]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank10]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc96528ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank10]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc96528ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank10]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc96528ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank10]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7fc917f5adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank30]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank30]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank30]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank30]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank30]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank30]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank30]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank30]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank30]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank30]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank10]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7fc917f674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank10]: frame #10: + 0x1205255 (0x7fc917f6d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank10]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7fc917f6e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank10]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7fc917f7f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank10]: frame #13: + 0x5fc28a2 (0x7fc9652828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank30]: frame #41: + 0x29d90 (0x7fc498e6ed90 in /lib/x86_64-linux-gnu/libc.so.6) [rank30]: frame #42: __libc_start_main + 0x80 (0x7fc498e6ee40 in /lib/x86_64-linux-gnu/libc.so.6) [rank30]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank30]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank10]: frame #14: + 0x5fcd035 (0x7fc96528d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank10]: frame #15: + 0x55d8ceb (0x7fc964898ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank27]: frame #16: + 0x55d6579 (0x7fbed3296579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank27]: frame #17: + 0x1aa7828 (0x7fbecf767828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank27]: frame #18: + 0x5fd6d74 (0x7fbed3c96d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank27]: frame #19: + 0x5fd7b05 (0x7fbed3c97b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank27]: frame #20: + 0xe1df28 (0x7fbee3783f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank27]: frame #21: + 0x4cf144 (0x7fbee2e35144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank11]: frame #16: + 0x55d6579 (0x7f1a5d296579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank11]: frame #17: + 0x1aa7828 (0x7f1a59767828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank11]: frame #18: + 0x5fd6d74 (0x7f1a5dc96d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank11]: frame #19: + 0x5fd7b05 (0x7f1a5dc97b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank11]: frame #20: + 0xe1df28 (0x7f1a6d783f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank11]: frame #21: + 0x4cf144 (0x7f1a6ce35144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank27]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank27]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank27]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank27]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank27]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank27]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank27]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank27]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank27]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank11]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank11]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank11]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank11]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank11]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank11]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank11]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank11]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank11]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank27]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank27]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank27]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank27]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank27]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank27]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank27]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank27]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank27]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank27]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank11]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank11]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank11]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank11]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank11]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank11]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank11]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank11]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank11]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank11]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank27]: frame #41: + 0x29d90 (0x7fbefa1f9d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank27]: frame #42: __libc_start_main + 0x80 (0x7fbefa1f9e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank27]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank27]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank11]: frame #41: + 0x29d90 (0x7f1a8414cd90 in /lib/x86_64-linux-gnu/libc.so.6) [rank11]: frame #42: __libc_start_main + 0x80 (0x7f1a8414ce40 in /lib/x86_64-linux-gnu/libc.so.6) [rank11]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank11]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank13]: Traceback (most recent call last): [rank13]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank13]: main() [rank13]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank13]: dist.barrier() [rank13]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank13]: return func(*args, **kwargs) [rank13]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank13]: work = group.barrier(opts=opts) [rank13]: torch.distributed.DistBackendError: [13] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank13]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank13]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fd9b16b8cf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank13]: frame #1: + 0x6021e58 (0x7fd9a12e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank13]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7fd9a12debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank13]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7fd9a12def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank13]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7fd9a12dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank13]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fd9a128ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank13]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fd9a128ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank13]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fd9a128ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank13]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7fd953f5adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank13]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7fd953f674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank13]: frame #10: + 0x1205255 (0x7fd953f6d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank13]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7fd953f6e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank13]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7fd953f7f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank13]: frame #13: + 0x5fc28a2 (0x7fd9a12828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank13]: frame #14: + 0x5fcd035 (0x7fd9a128d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank13]: frame #15: + 0x55d8ceb (0x7fd9a0898ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank14]: frame #16: + 0x55d6579 (0x7efdafc96579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank14]: frame #17: + 0x1aa7828 (0x7efdac167828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank14]: frame #18: + 0x5fd6d74 (0x7efdb0696d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank14]: frame #19: + 0x5fd7b05 (0x7efdb0697b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank14]: frame #20: + 0xe1df28 (0x7efdc0183f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank14]: frame #21: + 0x4cf144 (0x7efdbf835144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank14]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank14]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank14]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank14]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank14]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank14]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank14]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank14]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank14]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank14]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank14]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank14]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank14]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank14]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank14]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank14]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank14]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank14]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank14]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank14]: frame #41: + 0x29d90 (0x7efdd6ba8d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank14]: frame #42: __libc_start_main + 0x80 (0x7efdd6ba8e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank14]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank14]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank15]: frame #16: + 0x55d6579 (0x7f4662696579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank15]: frame #17: + 0x1aa7828 (0x7f465eb67828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank15]: frame #18: + 0x5fd6d74 (0x7f4663096d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank15]: frame #19: + 0x5fd7b05 (0x7f4663097b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank15]: frame #20: + 0xe1df28 (0x7f4672b83f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank15]: frame #21: + 0x4cf144 (0x7f4672235144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank15]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank15]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank15]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank15]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank15]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank15]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank15]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank15]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank15]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank15]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank15]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank15]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank15]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank15]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank15]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank15]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank15]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank15]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank15]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank15]: frame #41: + 0x29d90 (0x7f468952fd90 in /lib/x86_64-linux-gnu/libc.so.6) [rank15]: frame #42: __libc_start_main + 0x80 (0x7f468952fe40 in /lib/x86_64-linux-gnu/libc.so.6) [rank15]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank15]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank10]: frame #16: + 0x55d6579 (0x7fc964896579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank10]: frame #17: + 0x1aa7828 (0x7fc960d67828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank10]: frame #18: + 0x5fd6d74 (0x7fc965296d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank10]: frame #19: + 0x5fd7b05 (0x7fc965297b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank10]: frame #20: + 0xe1df28 (0x7fc974d83f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank10]: frame #21: + 0x4cf144 (0x7fc974435144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank10]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank10]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank10]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank10]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank10]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank10]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank10]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank10]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank10]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank10]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank10]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank10]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank10]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank10]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank10]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank10]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank10]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank10]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank10]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank10]: frame #41: + 0x29d90 (0x7fc98b721d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank10]: frame #42: __libc_start_main + 0x80 (0x7fc98b721e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank10]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank10]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank13]: frame #16: + 0x55d6579 (0x7fd9a0896579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank13]: frame #17: + 0x1aa7828 (0x7fd99cd67828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank13]: frame #18: + 0x5fd6d74 (0x7fd9a1296d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank13]: frame #19: + 0x5fd7b05 (0x7fd9a1297b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank13]: frame #20: + 0xe1df28 (0x7fd9b0d83f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank13]: frame #21: + 0x4cf144 (0x7fd9b0435144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank13]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank13]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank13]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank13]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank13]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank13]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank13]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank13]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank13]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank13]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank13]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank13]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank13]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank13]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank13]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank13]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank13]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank13]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank13]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank13]: frame #41: + 0x29d90 (0x7fd9c766fd90 in /lib/x86_64-linux-gnu/libc.so.6) [rank13]: frame #42: __libc_start_main + 0x80 (0x7fd9c766fe40 in /lib/x86_64-linux-gnu/libc.so.6) [rank13]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank13]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank58]: Traceback (most recent call last): [rank58]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank58]: main() [rank58]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank58]: dist.barrier() [rank58]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank58]: return func(*args, **kwargs) [rank58]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank58]: work = group.barrier(opts=opts) [rank58]: torch.distributed.DistBackendError: [58] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank58]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank33]: Traceback (most recent call last): [rank33]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank33]: main() [rank33]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank33]: dist.barrier() [rank33]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank33]: return func(*args, **kwargs) [rank33]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank33]: work = group.barrier(opts=opts) [rank33]: torch.distributed.DistBackendError: [33] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank33]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank33]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f8c4d0b8cf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank33]: frame #1: + 0x6021e58 (0x7f8c3cce1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank33]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f8c3ccdebdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank33]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f8c3ccdef5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank33]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f8c3ccdfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank58]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fea43b6bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank58]: frame #1: + 0x6021e58 (0x7fea336e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank58]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7fea336debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank58]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7fea336def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank58]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7fea336dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank33]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f8c3cc8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank33]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f8c3cc8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank33]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f8c3cc8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank33]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f8bef95adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank33]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f8bef9674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank33]: frame #10: + 0x1205255 (0x7f8bef96d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank33]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f8bef96e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank33]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f8bef97f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank33]: frame #13: + 0x5fc28a2 (0x7f8c3cc828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank33]: frame #14: + 0x5fcd035 (0x7f8c3cc8d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank33]: frame #15: + 0x55d8ceb (0x7f8c3c298ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank34]: Traceback (most recent call last): [rank34]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank34]: main() [rank34]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank34]: dist.barrier() [rank34]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank34]: return func(*args, **kwargs) [rank34]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank34]: work = group.barrier(opts=opts) [rank34]: torch.distributed.DistBackendError: [34] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank34]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank34]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f269076bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank34]: frame #1: + 0x6021e58 (0x7f26802e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank34]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f26802debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank34]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f26802def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank34]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f26802dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank34]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f268028ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank34]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f268028ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank34]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f268028ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank34]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f2632f5adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank34]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f2632f674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank34]: frame #10: + 0x1205255 (0x7f2632f6d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank34]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f2632f6e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank34]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f2632f7f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank34]: frame #13: + 0x5fc28a2 (0x7f26802828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank34]: frame #14: + 0x5fcd035 (0x7f268028d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank34]: frame #15: + 0x55d8ceb (0x7f267f898ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank35]: Traceback (most recent call last): [rank35]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank35]: main() [rank35]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank35]: dist.barrier() [rank35]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank35]: return func(*args, **kwargs) [rank35]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank35]: work = group.barrier(opts=opts) [rank35]: torch.distributed.DistBackendError: [35] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank35]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank35]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fbffaeb8cf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank35]: frame #1: + 0x6021e58 (0x7fbfeaae1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank35]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7fbfeaadebdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank35]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7fbfeaadef5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank35]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7fbfeaadfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank35]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fbfeaa8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank35]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fbfeaa8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank35]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fbfeaa8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank35]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7fbf9d75adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank58]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fea3368ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank58]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fea3368ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank58]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fea3368ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank58]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7fe9e635adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank35]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7fbf9d7674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank35]: frame #10: + 0x1205255 (0x7fbf9d76d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank35]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7fbf9d76e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank35]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7fbf9d77f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank35]: frame #13: + 0x5fc28a2 (0x7fbfeaa828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank58]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7fe9e63674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank58]: frame #10: + 0x1205255 (0x7fe9e636d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank58]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7fe9e636e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank58]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7fe9e637f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank58]: frame #13: + 0x5fc28a2 (0x7fea336828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank35]: frame #14: + 0x5fcd035 (0x7fbfeaa8d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank35]: frame #15: + 0x55d8ceb (0x7fbfea098ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank58]: frame #14: + 0x5fcd035 (0x7fea3368d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank58]: frame #15: + 0x55d8ceb (0x7fea32c98ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank36]: Traceback (most recent call last): [rank36]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank36]: main() [rank36]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank36]: dist.barrier() [rank36]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank36]: return func(*args, **kwargs) [rank36]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank36]: work = group.barrier(opts=opts) [rank36]: torch.distributed.DistBackendError: [36] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank36]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank59]: Traceback (most recent call last): [rank59]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank59]: main() [rank59]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank59]: dist.barrier() [rank59]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank59]: return func(*args, **kwargs) [rank59]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank59]: work = group.barrier(opts=opts) [rank59]: torch.distributed.DistBackendError: [59] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank59]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank36]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fa47296bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank36]: frame #1: + 0x6021e58 (0x7fa4624e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank36]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7fa4624debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank36]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7fa4624def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank36]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7fa4624dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank59]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f89a74d3cf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank59]: frame #1: + 0x6021e58 (0x7f89f5ae1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank59]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f89f5adebdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank59]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f89f5adef5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank59]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f89f5adfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank36]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fa46248ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank36]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fa46248ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank36]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fa46248ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank36]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7fa41515adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank59]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f89f5a8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank59]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f89f5a8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank59]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f89f5a8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank59]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f89a875adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank36]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7fa4151674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank36]: frame #10: + 0x1205255 (0x7fa41516d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank36]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7fa41516e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank36]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7fa41517f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank36]: frame #13: + 0x5fc28a2 (0x7fa4624828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank59]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f89a87674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank59]: frame #10: + 0x1205255 (0x7f89a876d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank59]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f89a876e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank59]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f89a877f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank59]: frame #13: + 0x5fc28a2 (0x7f89f5a828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank36]: frame #14: + 0x5fcd035 (0x7fa46248d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank36]: frame #15: + 0x55d8ceb (0x7fa461a98ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank59]: frame #14: + 0x5fcd035 (0x7f89f5a8d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank59]: frame #15: + 0x55d8ceb (0x7f89f5098ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank38]: Traceback (most recent call last): [rank38]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank38]: main() [rank38]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank38]: dist.barrier() [rank38]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank38]: return func(*args, **kwargs) [rank38]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank38]: work = group.barrier(opts=opts) [rank38]: torch.distributed.DistBackendError: [38] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank38]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank61]: Traceback (most recent call last): [rank61]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank61]: main() [rank61]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank61]: dist.barrier() [rank61]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank61]: return func(*args, **kwargs) [rank61]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank61]: work = group.barrier(opts=opts) [rank61]: torch.distributed.DistBackendError: [61] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank61]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank38]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fbe6b56bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank38]: frame #1: + 0x6021e58 (0x7fbe5b0e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank38]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7fbe5b0debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank38]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7fbe5b0def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank38]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7fbe5b0dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank61]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f8cf136bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank61]: frame #1: + 0x6021e58 (0x7f8ce0ee1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank61]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f8ce0edebdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank61]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f8ce0edef5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank61]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f8ce0edfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank38]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fbe5b08ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank38]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fbe5b08ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank38]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fbe5b08ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank38]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7fbe0dd5adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank61]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f8ce0e8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank61]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f8ce0e8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank61]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f8ce0e8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank61]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f8c93b5adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank38]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7fbe0dd674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank38]: frame #10: + 0x1205255 (0x7fbe0dd6d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank38]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7fbe0dd6e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank38]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7fbe0dd7f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank38]: frame #13: + 0x5fc28a2 (0x7fbe5b0828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank61]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f8c93b674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank61]: frame #10: + 0x1205255 (0x7f8c93b6d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank61]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f8c93b6e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank61]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f8c93b7f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank61]: frame #13: + 0x5fc28a2 (0x7f8ce0e828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank38]: frame #14: + 0x5fcd035 (0x7fbe5b08d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank38]: frame #15: + 0x55d8ceb (0x7fbe5a698ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank61]: frame #14: + 0x5fcd035 (0x7f8ce0e8d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank61]: frame #15: + 0x55d8ceb (0x7f8ce0498ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank32]: Traceback (most recent call last): [rank32]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank32]: main() [rank32]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank32]: dist.barrier() [rank32]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank32]: return func(*args, **kwargs) [rank32]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank32]: work = group.barrier(opts=opts) [rank32]: torch.distributed.DistBackendError: [32] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank32]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank63]: Traceback (most recent call last): [rank63]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank63]: main() [rank63]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank63]: dist.barrier() [rank63]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank63]: return func(*args, **kwargs) [rank63]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank63]: work = group.barrier(opts=opts) [rank63]: torch.distributed.DistBackendError: [63] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank63]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank32]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f5a95ab8cf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank32]: frame #1: + 0x6021e58 (0x7f5a856e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank32]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f5a856debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank32]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f5a856def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank32]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f5a856dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank63]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f76d00b8cf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank63]: frame #1: + 0x6021e58 (0x7f76bfce1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank63]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f76bfcdebdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank63]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f76bfcdef5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank63]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f76bfcdfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank32]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f5a8568ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank32]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f5a8568ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank32]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f5a8568ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank32]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f5a3835adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank63]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f76bfc8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank63]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f76bfc8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank63]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f76bfc8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank63]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f767295adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank32]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f5a383674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank32]: frame #10: + 0x1205255 (0x7f5a3836d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank32]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f5a3836e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank32]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f5a3837f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank32]: frame #13: + 0x5fc28a2 (0x7f5a856828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank63]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f76729674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank63]: frame #10: + 0x1205255 (0x7f767296d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank63]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f767296e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank63]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f767297f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank63]: frame #13: + 0x5fc28a2 (0x7f76bfc828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank32]: frame #14: + 0x5fcd035 (0x7f5a8568d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank32]: frame #15: + 0x55d8ceb (0x7f5a84c98ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank63]: frame #14: + 0x5fcd035 (0x7f76bfc8d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank63]: frame #15: + 0x55d8ceb (0x7f76bf298ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank33]: frame #16: + 0x55d6579 (0x7f8c3c296579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank33]: frame #17: + 0x1aa7828 (0x7f8c38767828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank33]: frame #18: + 0x5fd6d74 (0x7f8c3cc96d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank33]: frame #19: + 0x5fd7b05 (0x7f8c3cc97b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank33]: frame #20: + 0xe1df28 (0x7f8c4c783f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank33]: frame #21: + 0x4cf144 (0x7f8c4be35144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank56]: Traceback (most recent call last): [rank56]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank56]: main() [rank56]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank56]: dist.barrier() [rank56]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank56]: return func(*args, **kwargs) [rank56]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank56]: work = group.barrier(opts=opts) [rank56]: torch.distributed.DistBackendError: [56] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank56]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank33]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank33]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank33]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank33]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank33]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank33]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank33]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank33]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank33]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank56]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7eff7896bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank56]: frame #1: + 0x6021e58 (0x7eff684e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank56]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7eff684debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank56]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7eff684def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank56]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7eff684dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank33]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank33]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank33]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank33]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank33]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank33]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank33]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank33]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank33]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank33]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank56]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7eff6848ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank56]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7eff6848ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank56]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7eff6848ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank56]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7eff1b15adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank33]: frame #41: + 0x29d90 (0x7f8c630b6d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank33]: frame #42: __libc_start_main + 0x80 (0x7f8c630b6e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank33]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank33]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank56]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7eff1b1674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank56]: frame #10: + 0x1205255 (0x7eff1b16d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank56]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7eff1b16e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank56]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7eff1b17f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank56]: frame #13: + 0x5fc28a2 (0x7eff684828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank34]: frame #16: + 0x55d6579 (0x7f267f896579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank34]: frame #17: + 0x1aa7828 (0x7f267bd67828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank34]: frame #18: + 0x5fd6d74 (0x7f2680296d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank34]: frame #19: + 0x5fd7b05 (0x7f2680297b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank34]: frame #20: + 0xe1df28 (0x7f268fd83f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank34]: frame #21: + 0x4cf144 (0x7f268f435144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank56]: frame #14: + 0x5fcd035 (0x7eff6848d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank56]: frame #15: + 0x55d8ceb (0x7eff67a98ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank34]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank34]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank34]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank34]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank34]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank34]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank34]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank34]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank34]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank57]: Traceback (most recent call last): [rank57]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank57]: main() [rank57]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank57]: dist.barrier() [rank57]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank57]: return func(*args, **kwargs) [rank57]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank57]: work = group.barrier(opts=opts) [rank57]: torch.distributed.DistBackendError: [57] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank57]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank34]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank34]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank34]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank34]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank34]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank34]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank34]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank34]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank34]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank34]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank57]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f733416bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank57]: frame #1: + 0x6021e58 (0x7f7323ce1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank57]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f7323cdebdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank57]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f7323cdef5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank57]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f7323cdfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank34]: frame #41: + 0x29d90 (0x7f26a67e5d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank34]: frame #42: __libc_start_main + 0x80 (0x7f26a67e5e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank34]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank34]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank57]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f7323c8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank57]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f7323c8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank57]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f7323c8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank57]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f72d695adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank35]: frame #16: + 0x55d6579 (0x7fbfea096579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank35]: frame #17: + 0x1aa7828 (0x7fbfe6567828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank35]: frame #18: + 0x5fd6d74 (0x7fbfeaa96d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank35]: frame #19: + 0x5fd7b05 (0x7fbfeaa97b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank35]: frame #20: + 0xe1df28 (0x7fbffa583f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank35]: frame #21: + 0x4cf144 (0x7fbff9c35144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank57]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f72d69674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank57]: frame #10: + 0x1205255 (0x7f72d696d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank57]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f72d696e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank57]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f72d697f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank57]: frame #13: + 0x5fc28a2 (0x7f7323c828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank35]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank35]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank35]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank35]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank35]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank35]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank35]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank35]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank35]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank57]: frame #14: + 0x5fcd035 (0x7f7323c8d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank57]: frame #15: + 0x55d8ceb (0x7f7323298ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank35]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank35]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank35]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank35]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank35]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank35]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank35]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank35]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank35]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank35]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank58]: frame #16: + 0x55d6579 (0x7fea32c96579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank58]: frame #17: + 0x1aa7828 (0x7fea2f167828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank58]: frame #18: + 0x5fd6d74 (0x7fea33696d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank58]: frame #19: + 0x5fd7b05 (0x7fea33697b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank58]: frame #20: + 0xe1df28 (0x7fea43183f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank58]: frame #21: + 0x4cf144 (0x7fea42835144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank35]: frame #41: + 0x29d90 (0x7fc010ea6d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank35]: frame #42: __libc_start_main + 0x80 (0x7fc010ea6e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank35]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank35]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank58]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank58]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank58]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank58]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank58]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank58]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank58]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank58]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank58]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank36]: frame #16: + 0x55d6579 (0x7fa461a96579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank36]: frame #17: + 0x1aa7828 (0x7fa45df67828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank36]: frame #18: + 0x5fd6d74 (0x7fa462496d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank36]: frame #19: + 0x5fd7b05 (0x7fa462497b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank36]: frame #20: + 0xe1df28 (0x7fa471f83f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank36]: frame #21: + 0x4cf144 (0x7fa471635144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank58]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank58]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank58]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank58]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank58]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank58]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank58]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank58]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank58]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank58]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank36]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank36]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank36]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank36]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank36]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank36]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank36]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank36]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank36]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank58]: frame #41: + 0x29d90 (0x7fea59bced90 in /lib/x86_64-linux-gnu/libc.so.6) [rank58]: frame #42: __libc_start_main + 0x80 (0x7fea59bcee40 in /lib/x86_64-linux-gnu/libc.so.6) [rank58]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank58]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank36]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank36]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank36]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank36]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank36]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank36]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank36]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank36]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank36]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank36]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank59]: frame #16: + 0x55d6579 (0x7f89f5096579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank59]: frame #17: + 0x1aa7828 (0x7f89f1567828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank59]: frame #18: + 0x5fd6d74 (0x7f89f5a96d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank59]: frame #19: + 0x5fd7b05 (0x7f89f5a97b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank59]: frame #20: + 0xe1df28 (0x7f8a05583f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank59]: frame #21: + 0x4cf144 (0x7f8a04c35144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank36]: frame #41: + 0x29d90 (0x7fa488965d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank36]: frame #42: __libc_start_main + 0x80 (0x7fa488965e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank36]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank36]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank59]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank59]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank59]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank59]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank59]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank59]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank59]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank59]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank59]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank37]: Traceback (most recent call last): [rank37]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank37]: main() [rank37]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank37]: dist.barrier() [rank37]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank37]: return func(*args, **kwargs) [rank37]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank37]: work = group.barrier(opts=opts) [rank37]: torch.distributed.DistBackendError: [37] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank37]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank59]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank59]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank59]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank59]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank59]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank59]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank59]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank59]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank59]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank59]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank37]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fc71e56bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank37]: frame #1: + 0x6021e58 (0x7fc70e0e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank37]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7fc70e0debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank37]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7fc70e0def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank37]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7fc70e0dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank59]: frame #41: + 0x29d90 (0x7f8a1be36d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank59]: frame #42: __libc_start_main + 0x80 (0x7f8a1be36e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank59]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank59]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank37]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc70e08ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank37]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc70e08ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank37]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc70e08ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank37]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7fc6c0d5adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank60]: Traceback (most recent call last): [rank60]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank60]: main() [rank60]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank60]: dist.barrier() [rank60]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank60]: return func(*args, **kwargs) [rank60]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank60]: work = group.barrier(opts=opts) [rank60]: torch.distributed.DistBackendError: [60] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank60]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank37]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7fc6c0d674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank37]: frame #10: + 0x1205255 (0x7fc6c0d6d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank37]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7fc6c0d6e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank37]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7fc6c0d7f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank37]: frame #13: + 0x5fc28a2 (0x7fc70e0828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank60]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f4ca176bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank60]: frame #1: + 0x6021e58 (0x7f4c912e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank60]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f4c912debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank60]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f4c912def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank60]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f4c912dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank37]: frame #14: + 0x5fcd035 (0x7fc70e08d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank37]: frame #15: + 0x55d8ceb (0x7fc70d698ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank60]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f4c9128ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank60]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f4c9128ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank60]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f4c9128ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank60]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f4c43f5adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank38]: frame #16: + 0x55d6579 (0x7fbe5a696579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank38]: frame #17: + 0x1aa7828 (0x7fbe56b67828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank38]: frame #18: + 0x5fd6d74 (0x7fbe5b096d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank38]: frame #19: + 0x5fd7b05 (0x7fbe5b097b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank38]: frame #20: + 0xe1df28 (0x7fbe6ab83f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank38]: frame #21: + 0x4cf144 (0x7fbe6a235144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank60]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f4c43f674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank60]: frame #10: + 0x1205255 (0x7f4c43f6d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank60]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f4c43f6e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank60]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f4c43f7f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank60]: frame #13: + 0x5fc28a2 (0x7f4c912828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank38]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank38]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank38]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank38]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank38]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank38]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank38]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank38]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank38]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank60]: frame #14: + 0x5fcd035 (0x7f4c9128d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank60]: frame #15: + 0x55d8ceb (0x7f4c90898ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank38]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank38]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank38]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank38]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank38]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank38]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank38]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank38]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank38]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank38]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank61]: frame #16: + 0x55d6579 (0x7f8ce0496579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank61]: frame #17: + 0x1aa7828 (0x7f8cdc967828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank61]: frame #18: + 0x5fd6d74 (0x7f8ce0e96d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank61]: frame #19: + 0x5fd7b05 (0x7f8ce0e97b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank61]: frame #20: + 0xe1df28 (0x7f8cf0983f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank61]: frame #21: + 0x4cf144 (0x7f8cf0035144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank38]: frame #41: + 0x29d90 (0x7fbe8161dd90 in /lib/x86_64-linux-gnu/libc.so.6) [rank38]: frame #42: __libc_start_main + 0x80 (0x7fbe8161de40 in /lib/x86_64-linux-gnu/libc.so.6) [rank38]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank38]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank61]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank61]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank61]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank61]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank61]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank61]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank61]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank61]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank61]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank39]: Traceback (most recent call last): [rank39]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank39]: main() [rank39]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank39]: dist.barrier() [rank39]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank39]: return func(*args, **kwargs) [rank39]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank39]: work = group.barrier(opts=opts) [rank39]: torch.distributed.DistBackendError: [39] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank39]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank61]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank61]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank61]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank61]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank61]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank61]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank61]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank61]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank61]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank61]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank39]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f5b920b8cf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank39]: frame #1: + 0x6021e58 (0x7f5b81ce1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank39]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f5b81cdebdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank39]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f5b81cdef5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank39]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f5b81cdfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank61]: frame #41: + 0x29d90 (0x7f8d073a0d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank61]: frame #42: __libc_start_main + 0x80 (0x7f8d073a0e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank61]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank61]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank39]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f5b81c8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank39]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f5b81c8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank39]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f5b81c8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank39]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f5b3495adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank62]: Traceback (most recent call last): [rank62]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank62]: main() [rank62]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank62]: dist.barrier() [rank62]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank62]: return func(*args, **kwargs) [rank62]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank62]: work = group.barrier(opts=opts) [rank62]: torch.distributed.DistBackendError: [62] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank62]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank39]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f5b349674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank39]: frame #10: + 0x1205255 (0x7f5b3496d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank39]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f5b3496e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank39]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f5b3497f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank39]: frame #13: + 0x5fc28a2 (0x7f5b81c828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank62]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f3e3f96bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank62]: frame #1: + 0x6021e58 (0x7f3e2f4e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank62]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f3e2f4debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank62]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f3e2f4def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank62]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f3e2f4dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank39]: frame #14: + 0x5fcd035 (0x7f5b81c8d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank39]: frame #15: + 0x55d8ceb (0x7f5b81298ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank62]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f3e2f48ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank62]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f3e2f48ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank62]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f3e2f48ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank62]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f3de215adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank32]: frame #16: + 0x55d6579 (0x7f5a84c96579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank32]: frame #17: + 0x1aa7828 (0x7f5a81167828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank32]: frame #18: + 0x5fd6d74 (0x7f5a85696d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank32]: frame #19: + 0x5fd7b05 (0x7f5a85697b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank32]: frame #20: + 0xe1df28 (0x7f5a95183f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank32]: frame #21: + 0x4cf144 (0x7f5a94835144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank62]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f3de21674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank62]: frame #10: + 0x1205255 (0x7f3de216d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank62]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f3de216e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank62]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f3de217f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank62]: frame #13: + 0x5fc28a2 (0x7f3e2f4828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank32]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank32]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank32]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank32]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank32]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank32]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank32]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank32]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank32]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank62]: frame #14: + 0x5fcd035 (0x7f3e2f48d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank62]: frame #15: + 0x55d8ceb (0x7f3e2ea98ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank32]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank32]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank32]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank32]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank32]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank32]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank32]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank32]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank32]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank32]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank63]: frame #16: + 0x55d6579 (0x7f76bf296579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank63]: frame #17: + 0x1aa7828 (0x7f76bb767828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank63]: frame #18: + 0x5fd6d74 (0x7f76bfc96d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank63]: frame #19: + 0x5fd7b05 (0x7f76bfc97b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank63]: frame #20: + 0xe1df28 (0x7f76cf783f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank63]: frame #21: + 0x4cf144 (0x7f76cee35144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank32]: frame #41: + 0x29d90 (0x7f5aabab5d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank32]: frame #42: __libc_start_main + 0x80 (0x7f5aabab5e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank32]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank32]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank63]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank63]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank63]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank63]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank63]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank63]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank63]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank63]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank63]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank37]: frame #16: + 0x55d6579 (0x7fc70d696579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank37]: frame #17: + 0x1aa7828 (0x7fc709b67828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank37]: frame #18: + 0x5fd6d74 (0x7fc70e096d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank37]: frame #19: + 0x5fd7b05 (0x7fc70e097b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank37]: frame #20: + 0xe1df28 (0x7fc71db83f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank37]: frame #21: + 0x4cf144 (0x7fc71d235144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank63]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank63]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank63]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank63]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank63]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank63]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank63]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank63]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank63]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank63]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank37]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank37]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank37]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank37]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank37]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank37]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank37]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank37]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank37]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank63]: frame #41: + 0x29d90 (0x7f76e6101d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank63]: frame #42: __libc_start_main + 0x80 (0x7f76e6101e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank63]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank63]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank37]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank37]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank37]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank37]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank37]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank37]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank37]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank37]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank37]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank37]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank56]: frame #16: + 0x55d6579 (0x7eff67a96579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank56]: frame #17: + 0x1aa7828 (0x7eff63f67828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank56]: frame #18: + 0x5fd6d74 (0x7eff68496d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank56]: frame #19: + 0x5fd7b05 (0x7eff68497b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank56]: frame #20: + 0xe1df28 (0x7eff77f83f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank56]: frame #21: + 0x4cf144 (0x7eff77635144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank37]: frame #41: + 0x29d90 (0x7fc7345d9d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank37]: frame #42: __libc_start_main + 0x80 (0x7fc7345d9e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank37]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank37]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank56]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank56]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank56]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank56]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank56]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank56]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank56]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank56]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank56]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank39]: frame #16: + 0x55d6579 (0x7f5b81296579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank39]: frame #17: + 0x1aa7828 (0x7f5b7d767828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank39]: frame #18: + 0x5fd6d74 (0x7f5b81c96d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank39]: frame #19: + 0x5fd7b05 (0x7f5b81c97b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank39]: frame #20: + 0xe1df28 (0x7f5b91783f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank39]: frame #21: + 0x4cf144 (0x7f5b90e35144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank56]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank56]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank56]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank56]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank56]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank56]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank56]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank56]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank56]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank56]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank39]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank39]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank39]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank39]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank39]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank39]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank39]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank39]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank39]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank56]: frame #41: + 0x29d90 (0x7eff8e959d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank56]: frame #42: __libc_start_main + 0x80 (0x7eff8e959e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank56]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank56]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank39]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank39]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank39]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank39]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank39]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank39]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank39]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank39]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank39]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank39]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank57]: frame #16: + 0x55d6579 (0x7f7323296579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank57]: frame #17: + 0x1aa7828 (0x7f731f767828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank57]: frame #18: + 0x5fd6d74 (0x7f7323c96d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank57]: frame #19: + 0x5fd7b05 (0x7f7323c97b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank57]: frame #20: + 0xe1df28 (0x7f7333783f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank57]: frame #21: + 0x4cf144 (0x7f7332e35144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank39]: frame #41: + 0x29d90 (0x7f5ba80a8d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank39]: frame #42: __libc_start_main + 0x80 (0x7f5ba80a8e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank39]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank39]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank57]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank57]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank57]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank57]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank57]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank57]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank57]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank57]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank57]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank57]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank57]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank57]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank57]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank57]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank57]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank57]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank57]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank57]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank57]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank57]: frame #41: + 0x29d90 (0x7f734a206d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank57]: frame #42: __libc_start_main + 0x80 (0x7f734a206e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank57]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank57]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank60]: frame #16: + 0x55d6579 (0x7f4c90896579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank60]: frame #17: + 0x1aa7828 (0x7f4c8cd67828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank60]: frame #18: + 0x5fd6d74 (0x7f4c91296d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank60]: frame #19: + 0x5fd7b05 (0x7f4c91297b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank60]: frame #20: + 0xe1df28 (0x7f4ca0d83f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank60]: frame #21: + 0x4cf144 (0x7f4ca0435144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank60]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank60]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank60]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank60]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank60]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank60]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank60]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank60]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank60]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank60]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank60]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank60]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank60]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank60]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank60]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank60]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank60]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank60]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank60]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank60]: frame #41: + 0x29d90 (0x7f4cb77f4d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank60]: frame #42: __libc_start_main + 0x80 (0x7f4cb77f4e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank60]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank60]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank62]: frame #16: + 0x55d6579 (0x7f3e2ea96579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank62]: frame #17: + 0x1aa7828 (0x7f3e2af67828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank62]: frame #18: + 0x5fd6d74 (0x7f3e2f496d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank62]: frame #19: + 0x5fd7b05 (0x7f3e2f497b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank62]: frame #20: + 0xe1df28 (0x7f3e3ef83f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank62]: frame #21: + 0x4cf144 (0x7f3e3e635144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank62]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank62]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank62]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank62]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank62]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank62]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank62]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank62]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank62]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank62]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank62]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank62]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank62]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank62]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank62]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank62]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank62]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank62]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank62]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank62]: frame #41: + 0x29d90 (0x7f3e5593ad90 in /lib/x86_64-linux-gnu/libc.so.6) [rank62]: frame #42: __libc_start_main + 0x80 (0x7f3e5593ae40 in /lib/x86_64-linux-gnu/libc.so.6) [rank62]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank62]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank53]: Traceback (most recent call last): [rank53]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank53]: main() [rank53]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank53]: dist.barrier() [rank53]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank53]: return func(*args, **kwargs) [rank53]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank53]: work = group.barrier(opts=opts) [rank53]: torch.distributed.DistBackendError: [53] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank53]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank40]: Traceback (most recent call last): [rank40]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank40]: main() [rank40]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank40]: dist.barrier() [rank40]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank40]: return func(*args, **kwargs) [rank40]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank40]: work = group.barrier(opts=opts) [rank40]: torch.distributed.DistBackendError: [40] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank40]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank53]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fe218ab8cf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank53]: frame #1: + 0x6021e58 (0x7fe2086e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank53]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7fe2086debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank53]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7fe2086def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank53]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7fe2086dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank40]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f06ed6d3cf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank40]: frame #1: + 0x6021e58 (0x7f073bce1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank40]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f073bcdebdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank40]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f073bcdef5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank40]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f073bcdfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank40]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f073bc8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank40]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f073bc8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank40]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f073bc8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank40]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f06ee95adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank40]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f06ee9674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank40]: frame #10: + 0x1205255 (0x7f06ee96d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank40]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f06ee96e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank40]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f06ee97f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank40]: frame #13: + 0x5fc28a2 (0x7f073bc828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank40]: frame #14: + 0x5fcd035 (0x7f073bc8d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank40]: frame #15: + 0x55d8ceb (0x7f073b298ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank42]: Traceback (most recent call last): [rank42]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank42]: main() [rank42]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank42]: dist.barrier() [rank42]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank42]: return func(*args, **kwargs) [rank42]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank42]: work = group.barrier(opts=opts) [rank42]: torch.distributed.DistBackendError: [42] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank42]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank42]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fde310d3cf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank42]: frame #1: + 0x6021e58 (0x7fde7f6e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank42]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7fde7f6debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank42]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7fde7f6def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank42]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7fde7f6dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank42]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fde7f68ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank42]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fde7f68ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank42]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fde7f68ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank42]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7fde3235adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank42]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7fde323674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank42]: frame #10: + 0x1205255 (0x7fde3236d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank42]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7fde3236e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank42]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7fde3237f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank42]: frame #13: + 0x5fc28a2 (0x7fde7f6828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank42]: frame #14: + 0x5fcd035 (0x7fde7f68d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank42]: frame #15: + 0x55d8ceb (0x7fde7ec98ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank43]: Traceback (most recent call last): [rank43]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank43]: main() [rank43]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank43]: dist.barrier() [rank43]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank43]: return func(*args, **kwargs) [rank43]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank43]: work = group.barrier(opts=opts) [rank43]: torch.distributed.DistBackendError: [43] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank43]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank43]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f799fb6bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank43]: frame #1: + 0x6021e58 (0x7f798f6e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank43]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f798f6debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank43]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f798f6def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank43]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f798f6dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank43]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f798f68ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank43]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f798f68ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank43]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f798f68ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank43]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f794235adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank43]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f79423674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank43]: frame #10: + 0x1205255 (0x7f794236d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank43]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f794236e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank43]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f794237f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank43]: frame #13: + 0x5fc28a2 (0x7f798f6828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank43]: frame #14: + 0x5fcd035 (0x7f798f68d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank43]: frame #15: + 0x55d8ceb (0x7f798ec98ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank44]: Traceback (most recent call last): [rank44]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank44]: main() [rank44]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank44]: dist.barrier() [rank44]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank44]: return func(*args, **kwargs) [rank44]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank44]: work = group.barrier(opts=opts) [rank44]: torch.distributed.DistBackendError: [44] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank44]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank44]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f4c25cb8cf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank44]: frame #1: + 0x6021e58 (0x7f4c158e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank44]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f4c158debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank44]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f4c158def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank44]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f4c158dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank44]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f4c1588ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank44]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f4c1588ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank44]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f4c1588ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank44]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f4bc855adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank44]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f4bc85674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank44]: frame #10: + 0x1205255 (0x7f4bc856d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank44]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f4bc856e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank44]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f4bc857f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank44]: frame #13: + 0x5fc28a2 (0x7f4c158828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank44]: frame #14: + 0x5fcd035 (0x7f4c1588d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank44]: frame #15: + 0x55d8ceb (0x7f4c14e98ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank46]: Traceback (most recent call last): [rank46]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank46]: main() [rank46]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank46]: dist.barrier() [rank46]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank46]: return func(*args, **kwargs) [rank46]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank46]: work = group.barrier(opts=opts) [rank46]: torch.distributed.DistBackendError: [46] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank46]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank46]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f4cad56bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank46]: frame #1: + 0x6021e58 (0x7f4c9d0e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank46]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f4c9d0debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank46]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f4c9d0def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank46]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f4c9d0dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank46]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f4c9d08ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank46]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f4c9d08ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank46]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f4c9d08ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank46]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f4c4fd5adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank46]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f4c4fd674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank46]: frame #10: + 0x1205255 (0x7f4c4fd6d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank46]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f4c4fd6e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank46]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f4c4fd7f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank46]: frame #13: + 0x5fc28a2 (0x7f4c9d0828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank46]: frame #14: + 0x5fcd035 (0x7f4c9d08d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank46]: frame #15: + 0x55d8ceb (0x7f4c9c698ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank47]: Traceback (most recent call last): [rank47]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank47]: main() [rank47]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank47]: dist.barrier() [rank47]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank47]: return func(*args, **kwargs) [rank47]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank47]: work = group.barrier(opts=opts) [rank47]: torch.distributed.DistBackendError: [47] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank47]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank47]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fbd4696bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank47]: frame #1: + 0x6021e58 (0x7fbd364e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank47]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7fbd364debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank47]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7fbd364def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank47]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7fbd364dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank47]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fbd3648ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank47]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fbd3648ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank47]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fbd3648ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank47]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7fbce915adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank47]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7fbce91674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank47]: frame #10: + 0x1205255 (0x7fbce916d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank47]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7fbce916e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank47]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7fbce917f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank47]: frame #13: + 0x5fc28a2 (0x7fbd364828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank53]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fe20868ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank53]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fe20868ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank53]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fe20868ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank53]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7fe1bb35adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank47]: frame #14: + 0x5fcd035 (0x7fbd3648d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank47]: frame #15: + 0x55d8ceb (0x7fbd35a98ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank53]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7fe1bb3674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank53]: frame #10: + 0x1205255 (0x7fe1bb36d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank53]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7fe1bb36e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank53]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7fe1bb37f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank53]: frame #13: + 0x5fc28a2 (0x7fe2086828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank40]: frame #16: + 0x55d6579 (0x7f073b296579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank40]: frame #17: + 0x1aa7828 (0x7f0737767828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank40]: frame #18: + 0x5fd6d74 (0x7f073bc96d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank40]: frame #19: + 0x5fd7b05 (0x7f073bc97b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank40]: frame #20: + 0xe1df28 (0x7f074b783f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank40]: frame #21: + 0x4cf144 (0x7f074ae35144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank53]: frame #14: + 0x5fcd035 (0x7fe20868d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank53]: frame #15: + 0x55d8ceb (0x7fe207c98ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank40]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank40]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank40]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank40]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank40]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank40]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank40]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank40]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank40]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank48]: Traceback (most recent call last): [rank48]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank48]: main() [rank48]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank48]: dist.barrier() [rank48]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank48]: return func(*args, **kwargs) [rank48]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank48]: work = group.barrier(opts=opts) [rank48]: torch.distributed.DistBackendError: [48] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank48]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank40]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank40]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank40]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank40]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank40]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank40]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank40]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank40]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank40]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank40]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank48]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fac1b6b8cf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank48]: frame #1: + 0x6021e58 (0x7fac0b2e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank48]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7fac0b2debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank48]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7fac0b2def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank48]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7fac0b2dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank40]: frame #41: + 0x29d90 (0x7f0762038d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank40]: frame #42: __libc_start_main + 0x80 (0x7f0762038e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank40]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank40]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank48]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fac0b28ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank48]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fac0b28ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank48]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fac0b28ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank48]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7fabbdf5adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank41]: Traceback (most recent call last): [rank41]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank41]: main() [rank41]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank41]: dist.barrier() [rank41]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank41]: return func(*args, **kwargs) [rank41]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank41]: work = group.barrier(opts=opts) [rank41]: torch.distributed.DistBackendError: [41] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank41]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank48]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7fabbdf674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank48]: frame #10: + 0x1205255 (0x7fabbdf6d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank48]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7fabbdf6e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank48]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7fabbdf7f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank48]: frame #13: + 0x5fc28a2 (0x7fac0b2828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank41]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fcace2d3cf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank41]: frame #1: + 0x6021e58 (0x7fcb1c8e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank41]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7fcb1c8debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank41]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7fcb1c8def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank41]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7fcb1c8dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank48]: frame #14: + 0x5fcd035 (0x7fac0b28d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank48]: frame #15: + 0x55d8ceb (0x7fac0a898ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank41]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fcb1c88ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank41]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fcb1c88ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank41]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fcb1c88ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank41]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7fcacf55adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank49]: Traceback (most recent call last): [rank49]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank49]: main() [rank49]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank49]: dist.barrier() [rank49]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank49]: return func(*args, **kwargs) [rank49]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank49]: work = group.barrier(opts=opts) [rank49]: torch.distributed.DistBackendError: [49] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank49]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank41]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7fcacf5674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank41]: frame #10: + 0x1205255 (0x7fcacf56d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank41]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7fcacf56e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank41]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7fcacf57f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank41]: frame #13: + 0x5fc28a2 (0x7fcb1c8828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank49]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f269c2b8cf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank49]: frame #1: + 0x6021e58 (0x7f268bee1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank49]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f268bedebdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank49]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f268bedef5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank49]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f268bedfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank41]: frame #14: + 0x5fcd035 (0x7fcb1c88d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank41]: frame #15: + 0x55d8ceb (0x7fcb1be98ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank49]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f268be8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank49]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f268be8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank49]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f268be8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank49]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f263eb5adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank42]: frame #16: + 0x55d6579 (0x7fde7ec96579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank42]: frame #17: + 0x1aa7828 (0x7fde7b167828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank42]: frame #18: + 0x5fd6d74 (0x7fde7f696d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank42]: frame #19: + 0x5fd7b05 (0x7fde7f697b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank42]: frame #20: + 0xe1df28 (0x7fde8f183f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank42]: frame #21: + 0x4cf144 (0x7fde8e835144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank49]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f263eb674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank49]: frame #10: + 0x1205255 (0x7f263eb6d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank49]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f263eb6e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank49]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f263eb7f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank49]: frame #13: + 0x5fc28a2 (0x7f268be828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank42]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank42]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank42]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank42]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank42]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank42]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank42]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank42]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank42]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank49]: frame #14: + 0x5fcd035 (0x7f268be8d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank49]: frame #15: + 0x55d8ceb (0x7f268b498ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank42]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank42]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank42]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank42]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank42]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank42]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank42]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank42]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank42]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank42]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank50]: Traceback (most recent call last): [rank50]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank50]: main() [rank50]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank50]: dist.barrier() [rank50]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank50]: return func(*args, **kwargs) [rank50]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank50]: work = group.barrier(opts=opts) [rank50]: torch.distributed.DistBackendError: [50] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank50]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank42]: frame #41: + 0x29d90 (0x7fdea5a23d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank42]: frame #42: __libc_start_main + 0x80 (0x7fdea5a23e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank42]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank42]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank50]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f38a736bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank50]: frame #1: + 0x6021e58 (0x7f3896ee1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank50]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f3896edebdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank50]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f3896edef5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank50]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f3896edfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank43]: frame #16: + 0x55d6579 (0x7f798ec96579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank43]: frame #17: + 0x1aa7828 (0x7f798b167828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank43]: frame #18: + 0x5fd6d74 (0x7f798f696d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank43]: frame #19: + 0x5fd7b05 (0x7f798f697b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank43]: frame #20: + 0xe1df28 (0x7f799f183f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank43]: frame #21: + 0x4cf144 (0x7f799e835144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank50]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f3896e8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank50]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f3896e8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank50]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f3896e8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank50]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f3849b5adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank43]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank43]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank43]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank43]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank43]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank43]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank43]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank43]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank43]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank50]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f3849b674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank50]: frame #10: + 0x1205255 (0x7f3849b6d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank50]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f3849b6e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank50]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f3849b7f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank50]: frame #13: + 0x5fc28a2 (0x7f3896e828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank43]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank43]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank43]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank43]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank43]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank43]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank43]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank43]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank43]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank43]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank50]: frame #14: + 0x5fcd035 (0x7f3896e8d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank50]: frame #15: + 0x55d8ceb (0x7f3896498ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank43]: frame #41: + 0x29d90 (0x7f79b5b2ad90 in /lib/x86_64-linux-gnu/libc.so.6) [rank43]: frame #42: __libc_start_main + 0x80 (0x7f79b5b2ae40 in /lib/x86_64-linux-gnu/libc.so.6) [rank43]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank43]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank51]: Traceback (most recent call last): [rank51]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank51]: main() [rank51]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank51]: dist.barrier() [rank51]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank51]: return func(*args, **kwargs) [rank51]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank51]: work = group.barrier(opts=opts) [rank51]: torch.distributed.DistBackendError: [51] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank51]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank44]: frame #16: + 0x55d6579 (0x7f4c14e96579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank44]: frame #17: + 0x1aa7828 (0x7f4c11367828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank44]: frame #18: + 0x5fd6d74 (0x7f4c15896d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank44]: frame #19: + 0x5fd7b05 (0x7f4c15897b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank44]: frame #20: + 0xe1df28 (0x7f4c25383f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank44]: frame #21: + 0x4cf144 (0x7f4c24a35144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank51]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f2ef4d6bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank51]: frame #1: + 0x6021e58 (0x7f2ee48e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank51]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f2ee48debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank51]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f2ee48def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank51]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f2ee48dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank44]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank44]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank44]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank44]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank44]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank44]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank44]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank44]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank44]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank51]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f2ee488ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank51]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f2ee488ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank51]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f2ee488ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank51]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f2e9755adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank44]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank44]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank44]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank44]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank44]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank44]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank44]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank44]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank44]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank44]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank51]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f2e975674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank51]: frame #10: + 0x1205255 (0x7f2e9756d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank51]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f2e9756e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank51]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f2e9757f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank51]: frame #13: + 0x5fc28a2 (0x7f2ee48828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank44]: frame #41: + 0x29d90 (0x7f4c3bcf1d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank44]: frame #42: __libc_start_main + 0x80 (0x7f4c3bcf1e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank44]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank44]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank51]: frame #14: + 0x5fcd035 (0x7f2ee488d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank51]: frame #15: + 0x55d8ceb (0x7f2ee3e98ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank45]: Traceback (most recent call last): [rank45]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank45]: main() [rank45]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank45]: dist.barrier() [rank45]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank45]: return func(*args, **kwargs) [rank45]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank45]: work = group.barrier(opts=opts) [rank45]: torch.distributed.DistBackendError: [45] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank45]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank52]: Traceback (most recent call last): [rank52]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank52]: main() [rank52]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank52]: dist.barrier() [rank52]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank52]: return func(*args, **kwargs) [rank52]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank52]: work = group.barrier(opts=opts) [rank52]: torch.distributed.DistBackendError: [52] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank52]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank45]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f30fb2b8cf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank45]: frame #1: + 0x6021e58 (0x7f30eaee1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank45]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f30eaedebdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank45]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f30eaedef5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank45]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f30eaedfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank52]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fddbaab8cf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank52]: frame #1: + 0x6021e58 (0x7fddaa6e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank52]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7fddaa6debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank52]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7fddaa6def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank52]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7fddaa6dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank45]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f30eae8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank45]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f30eae8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank45]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f30eae8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank45]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f309db5adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank52]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fddaa68ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank52]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fddaa68ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank52]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fddaa68ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank52]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7fdd5d35adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank45]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f309db674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank45]: frame #10: + 0x1205255 (0x7f309db6d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank45]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f309db6e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank45]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f309db7f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank45]: frame #13: + 0x5fc28a2 (0x7f30eae828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank52]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7fdd5d3674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank52]: frame #10: + 0x1205255 (0x7fdd5d36d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank52]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7fdd5d36e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank52]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7fdd5d37f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank52]: frame #13: + 0x5fc28a2 (0x7fddaa6828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank45]: frame #14: + 0x5fcd035 (0x7f30eae8d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank45]: frame #15: + 0x55d8ceb (0x7f30ea498ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank52]: frame #14: + 0x5fcd035 (0x7fddaa68d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank52]: frame #15: + 0x55d8ceb (0x7fdda9c98ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank46]: frame #16: + 0x55d6579 (0x7f4c9c696579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank46]: frame #17: + 0x1aa7828 (0x7f4c98b67828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank46]: frame #18: + 0x5fd6d74 (0x7f4c9d096d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank46]: frame #19: + 0x5fd7b05 (0x7f4c9d097b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank46]: frame #20: + 0xe1df28 (0x7f4cacb83f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank46]: frame #21: + 0x4cf144 (0x7f4cac235144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank53]: frame #16: + 0x55d6579 (0x7fe207c96579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank53]: frame #17: + 0x1aa7828 (0x7fe204167828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank53]: frame #18: + 0x5fd6d74 (0x7fe208696d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank53]: frame #19: + 0x5fd7b05 (0x7fe208697b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank53]: frame #20: + 0xe1df28 (0x7fe218183f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank53]: frame #21: + 0x4cf144 (0x7fe217835144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank46]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank46]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank46]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank46]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank46]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank46]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank46]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank46]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank46]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank53]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank53]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank53]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank53]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank53]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank53]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank53]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank53]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank53]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank46]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank46]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank46]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank46]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank46]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank46]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank46]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank46]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank46]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank46]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank53]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank53]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank53]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank53]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank53]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank53]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank53]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank53]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank53]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank53]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank46]: frame #41: + 0x29d90 (0x7f4cc35e9d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank46]: frame #42: __libc_start_main + 0x80 (0x7f4cc35e9e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank46]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank46]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank53]: frame #41: + 0x29d90 (0x7fe22eb0dd90 in /lib/x86_64-linux-gnu/libc.so.6) [rank53]: frame #42: __libc_start_main + 0x80 (0x7fe22eb0de40 in /lib/x86_64-linux-gnu/libc.so.6) [rank53]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank53]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank47]: frame #16: + 0x55d6579 (0x7fbd35a96579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank47]: frame #17: + 0x1aa7828 (0x7fbd31f67828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank47]: frame #18: + 0x5fd6d74 (0x7fbd36496d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank47]: frame #19: + 0x5fd7b05 (0x7fbd36497b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank47]: frame #20: + 0xe1df28 (0x7fbd45f83f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank47]: frame #21: + 0x4cf144 (0x7fbd45635144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank54]: Traceback (most recent call last): [rank54]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank54]: main() [rank54]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank54]: dist.barrier() [rank54]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank54]: return func(*args, **kwargs) [rank54]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank54]: work = group.barrier(opts=opts) [rank54]: torch.distributed.DistBackendError: [54] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank54]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank47]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank47]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank47]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank47]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank47]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank47]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank47]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank47]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank47]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank54]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fc08c96bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank54]: frame #1: + 0x6021e58 (0x7fc07c4e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank54]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7fc07c4debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank54]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7fc07c4def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank54]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7fc07c4dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank47]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank47]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank47]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank47]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank47]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank47]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank47]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank47]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank47]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank47]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank54]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc07c48ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank54]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc07c48ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank54]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc07c48ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank54]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7fc02f15adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank47]: frame #41: + 0x29d90 (0x7fbd5c977d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank47]: frame #42: __libc_start_main + 0x80 (0x7fbd5c977e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank47]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank47]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank54]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7fc02f1674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank54]: frame #10: + 0x1205255 (0x7fc02f16d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank54]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7fc02f16e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank54]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7fc02f17f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank54]: frame #13: + 0x5fc28a2 (0x7fc07c4828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank41]: frame #16: + 0x55d6579 (0x7fcb1be96579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank41]: frame #17: + 0x1aa7828 (0x7fcb18367828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank41]: frame #18: + 0x5fd6d74 (0x7fcb1c896d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank41]: frame #19: + 0x5fd7b05 (0x7fcb1c897b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank41]: frame #20: + 0xe1df28 (0x7fcb2c383f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank41]: frame #21: + 0x4cf144 (0x7fcb2ba35144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank54]: frame #14: + 0x5fcd035 (0x7fc07c48d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank54]: frame #15: + 0x55d8ceb (0x7fc07ba98ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank41]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank41]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank41]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank41]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank41]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank41]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank41]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank41]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank41]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank55]: Traceback (most recent call last): [rank55]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank55]: main() [rank55]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank55]: dist.barrier() [rank55]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank55]: return func(*args, **kwargs) [rank55]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank55]: work = group.barrier(opts=opts) [rank55]: torch.distributed.DistBackendError: [55] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank55]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank41]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank41]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank41]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank41]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank41]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank41]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank41]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank41]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank41]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank41]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank55]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f0da36d3cf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank55]: frame #1: + 0x6021e58 (0x7f0df1ce1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank55]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f0df1cdebdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank55]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f0df1cdef5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank55]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f0df1cdfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank41]: frame #41: + 0x29d90 (0x7fcb42c3ad90 in /lib/x86_64-linux-gnu/libc.so.6) [rank41]: frame #42: __libc_start_main + 0x80 (0x7fcb42c3ae40 in /lib/x86_64-linux-gnu/libc.so.6) [rank41]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank41]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank55]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f0df1c8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank55]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f0df1c8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank55]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f0df1c8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank55]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f0da495adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank45]: frame #16: + 0x55d6579 (0x7f30ea496579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank45]: frame #17: + 0x1aa7828 (0x7f30e6967828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank45]: frame #18: + 0x5fd6d74 (0x7f30eae96d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank45]: frame #19: + 0x5fd7b05 (0x7f30eae97b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank45]: frame #20: + 0xe1df28 (0x7f30fa983f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank45]: frame #21: + 0x4cf144 (0x7f30fa035144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank55]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f0da49674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank55]: frame #10: + 0x1205255 (0x7f0da496d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank55]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f0da496e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank55]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f0da497f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank55]: frame #13: + 0x5fc28a2 (0x7f0df1c828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank45]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank45]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank45]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank45]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank45]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank45]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank45]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank45]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank45]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank55]: frame #14: + 0x5fcd035 (0x7f0df1c8d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank55]: frame #15: + 0x55d8ceb (0x7f0df1298ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank45]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank45]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank45]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank45]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank45]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank45]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank45]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank45]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank45]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank45]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank48]: frame #16: + 0x55d6579 (0x7fac0a896579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank48]: frame #17: + 0x1aa7828 (0x7fac06d67828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank48]: frame #18: + 0x5fd6d74 (0x7fac0b296d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank48]: frame #19: + 0x5fd7b05 (0x7fac0b297b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank48]: frame #20: + 0xe1df28 (0x7fac1ad83f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank48]: frame #21: + 0x4cf144 (0x7fac1a435144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank45]: frame #41: + 0x29d90 (0x7f31112b6d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank45]: frame #42: __libc_start_main + 0x80 (0x7f31112b6e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank45]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank45]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank48]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank48]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank48]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank48]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank48]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank48]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank48]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank48]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank48]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank48]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank48]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank48]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank48]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank48]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank48]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank48]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank48]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank48]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank48]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank48]: frame #41: + 0x29d90 (0x7fac316b4d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank48]: frame #42: __libc_start_main + 0x80 (0x7fac316b4e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank48]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank48]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank49]: frame #16: + 0x55d6579 (0x7f268b496579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank49]: frame #17: + 0x1aa7828 (0x7f2687967828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank49]: frame #18: + 0x5fd6d74 (0x7f268be96d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank49]: frame #19: + 0x5fd7b05 (0x7f268be97b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank49]: frame #20: + 0xe1df28 (0x7f269b983f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank49]: frame #21: + 0x4cf144 (0x7f269b035144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank49]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank49]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank49]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank49]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank49]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank49]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank49]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank49]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank49]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank49]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank49]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank49]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank49]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank49]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank49]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank49]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank49]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank49]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank49]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank49]: frame #41: + 0x29d90 (0x7f26b22d2d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank49]: frame #42: __libc_start_main + 0x80 (0x7f26b22d2e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank49]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank49]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank50]: frame #16: + 0x55d6579 (0x7f3896496579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank50]: frame #17: + 0x1aa7828 (0x7f3892967828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank50]: frame #18: + 0x5fd6d74 (0x7f3896e96d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank50]: frame #19: + 0x5fd7b05 (0x7f3896e97b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank50]: frame #20: + 0xe1df28 (0x7f38a6983f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank50]: frame #21: + 0x4cf144 (0x7f38a6035144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank50]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank50]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank50]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank50]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank50]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank50]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank50]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank50]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank50]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank50]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank50]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank50]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank50]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank50]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank50]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank50]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank50]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank50]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank50]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank50]: frame #41: + 0x29d90 (0x7f38bd3bfd90 in /lib/x86_64-linux-gnu/libc.so.6) [rank50]: frame #42: __libc_start_main + 0x80 (0x7f38bd3bfe40 in /lib/x86_64-linux-gnu/libc.so.6) [rank50]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank50]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank51]: frame #16: + 0x55d6579 (0x7f2ee3e96579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank51]: frame #17: + 0x1aa7828 (0x7f2ee0367828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank51]: frame #18: + 0x5fd6d74 (0x7f2ee4896d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank51]: frame #19: + 0x5fd7b05 (0x7f2ee4897b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank51]: frame #20: + 0xe1df28 (0x7f2ef4383f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank51]: frame #21: + 0x4cf144 (0x7f2ef3a35144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank51]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank51]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank51]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank51]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank51]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank51]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank51]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank51]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank51]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank51]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank51]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank51]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank51]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank51]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank51]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank51]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank51]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank51]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank51]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank51]: frame #41: + 0x29d90 (0x7f2f0ade8d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank51]: frame #42: __libc_start_main + 0x80 (0x7f2f0ade8e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank51]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank51]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank52]: frame #16: + 0x55d6579 (0x7fdda9c96579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank52]: frame #17: + 0x1aa7828 (0x7fdda6167828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank52]: frame #18: + 0x5fd6d74 (0x7fddaa696d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank52]: frame #19: + 0x5fd7b05 (0x7fddaa697b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank52]: frame #20: + 0xe1df28 (0x7fddba183f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank52]: frame #21: + 0x4cf144 (0x7fddb9835144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank52]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank52]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank52]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank52]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank52]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank52]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank52]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank52]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank52]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank52]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank52]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank52]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank52]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank52]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank52]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank52]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank52]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank52]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank52]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank52]: frame #41: + 0x29d90 (0x7fddd0b04d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank52]: frame #42: __libc_start_main + 0x80 (0x7fddd0b04e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank52]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank52]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank54]: frame #16: + 0x55d6579 (0x7fc07ba96579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank54]: frame #17: + 0x1aa7828 (0x7fc077f67828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank54]: frame #18: + 0x5fd6d74 (0x7fc07c496d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank54]: frame #19: + 0x5fd7b05 (0x7fc07c497b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank54]: frame #20: + 0xe1df28 (0x7fc08bf83f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank54]: frame #21: + 0x4cf144 (0x7fc08b635144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank54]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank54]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank54]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank54]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank54]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank54]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank54]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank54]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank54]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank54]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank54]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank54]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank54]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank54]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank54]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank54]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank54]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank54]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank54]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank54]: frame #41: + 0x29d90 (0x7fc0a296fd90 in /lib/x86_64-linux-gnu/libc.so.6) [rank54]: frame #42: __libc_start_main + 0x80 (0x7fc0a296fe40 in /lib/x86_64-linux-gnu/libc.so.6) [rank54]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank54]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank55]: frame #16: + 0x55d6579 (0x7f0df1296579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank55]: frame #17: + 0x1aa7828 (0x7f0ded767828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank55]: frame #18: + 0x5fd6d74 (0x7f0df1c96d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank55]: frame #19: + 0x5fd7b05 (0x7f0df1c97b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank55]: frame #20: + 0xe1df28 (0x7f0e01783f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank55]: frame #21: + 0x4cf144 (0x7f0e00e35144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank55]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank55]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank55]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank55]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank55]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank55]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank55]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank55]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank55]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank55]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank55]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank55]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank55]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank55]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank55]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank55]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank55]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank55]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank55]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank55]: frame #41: + 0x29d90 (0x7f0e18049d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank55]: frame #42: __libc_start_main + 0x80 (0x7f0e18049e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank55]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank55]: . This may indicate a possible application crash on rank 0 or a network set up issue. srun: error: h100-ord01-03-592: task 0: Exited with exit code 1 [rank8]:[W1114 23:18:15.117085519 ProcessGroupNCCL.cpp:1326] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator()) [rank24]:[W1114 23:18:15.533000069 ProcessGroupNCCL.cpp:1326] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator()) [rank16]:[W1114 23:18:15.762228382 ProcessGroupNCCL.cpp:1326] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator()) [rank40]:[W1114 23:18:15.004365378 ProcessGroupNCCL.cpp:1326] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator()) [rank56]:[W1114 23:18:15.846521844 ProcessGroupNCCL.cpp:1326] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator()) [rank48]:[W1114 23:18:15.323076082 ProcessGroupNCCL.cpp:1326] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator()) [rank32]:[W1114 23:18:15.515571718 ProcessGroupNCCL.cpp:1326] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator()) srun: error: h100-ord01-03-592: task 4: Exited with exit code 1 srun: error: h100-ord01-03-710: tasks 21,23: Exited with exit code 1 srun: error: h100-ord01-03-507: tasks 8,14: Exited with exit code 1 srun: error: h100-ord01-03-603: tasks 28,30: Exited with exit code 1 srun: error: h100-ord01-03-592: tasks 5,7: Exited with exit code 1 srun: error: h100-ord01-03-507: tasks 9,12: Exited with exit code 1 srun: error: h100-ord01-03-710: tasks 17,20: Exited with exit code 1 srun: error: h100-ord01-03-603: tasks 24-25: Exited with exit code 1 srun: error: h100-ord01-03-710: tasks 18,22: Exited with exit code 1 srun: error: h100-ord01-03-507: tasks 10,15: Exited with exit code 1 srun: error: h100-ord01-03-603: tasks 26,31: Exited with exit code 1 srun: error: h100-ord01-03-710: task 16: Exited with exit code 1 srun: error: h100-ord01-03-507: task 11: Exited with exit code 1 srun: error: h100-ord01-03-603: task 27: Exited with exit code 1 srun: error: h100-ord01-03-507: task 13: Exited with exit code 1 srun: error: h100-ord01-03-710: task 19: Exited with exit code 1 srun: error: h100-ord01-03-603: task 29: Exited with exit code 1 srun: error: h100-ord01-03-979: tasks 42,44: Exited with exit code 1 srun: error: h100-ord01-03-735: tasks 33,38: Exited with exit code 1 srun: error: h100-ord01-03-867: tasks 56,61: Exited with exit code 1 srun: error: h100-ord01-03-637: tasks 50,54: Exited with exit code 1 srun: error: h100-ord01-03-867: tasks 58,60: Exited with exit code 1 srun: error: h100-ord01-03-979: tasks 43,46: Exited with exit code 1 srun: error: h100-ord01-03-637: tasks 52,55: Exited with exit code 1 srun: error: h100-ord01-03-735: tasks 35,39: Exited with exit code 1 srun: error: h100-ord01-03-979: task 45: Exited with exit code 1 srun: error: h100-ord01-03-867: tasks 57,59: Exited with exit code 1 srun: error: h100-ord01-03-637: task 48: Exited with exit code 1 srun: error: h100-ord01-03-735: tasks 32,34,36: Exited with exit code 1 srun: error: h100-ord01-03-979: tasks 41,47: Exited with exit code 1 srun: error: h100-ord01-03-867: task 62: Exited with exit code 1 srun: error: h100-ord01-03-637: tasks 49,51: Exited with exit code 1 srun: error: h100-ord01-03-979: task 40: Exited with exit code 1 srun: error: h100-ord01-03-867: task 63: Exited with exit code 1 srun: error: h100-ord01-03-735: task 37: Exited with exit code 1 srun: error: h100-ord01-03-637: task 53: Exited with exit code 1 [rank6]:[W1114 23:18:24.161644180 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 6] using GPU 6 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank6]: Traceback (most recent call last): [rank6]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank6]: main() [rank6]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank6]: dist.barrier() [rank6]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank6]: return func(*args, **kwargs) [rank6]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank6]: work = group.barrier(opts=opts) [rank6]: torch.distributed.DistBackendError: [6] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank6]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank6]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7efd914b8cf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank6]: frame #1: + 0x6021e58 (0x7efd810e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank6]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7efd810debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank6]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7efd810def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank6]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7efd810dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank6]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7efd8108ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank6]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7efd8108ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank6]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7efd8108ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank6]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7efd33d5adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank6]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7efd33d674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank6]: frame #10: + 0x1205255 (0x7efd33d6d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank6]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7efd33d6e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank6]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7efd33d7f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank6]: frame #13: + 0x5fc28a2 (0x7efd810828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank6]: frame #14: + 0x5fcd035 (0x7efd8108d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank6]: frame #15: + 0x55d8ceb (0x7efd80698ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank6]: frame #16: + 0x55d6579 (0x7efd80696579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank6]: frame #17: + 0x1aa7828 (0x7efd7cb67828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank6]: frame #18: + 0x5fd6d74 (0x7efd81096d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank6]: frame #19: + 0x5fd7b05 (0x7efd81097b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank6]: frame #20: + 0xe1df28 (0x7efd90b83f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank6]: frame #21: + 0x4cf144 (0x7efd90235144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank6]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank6]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank6]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank6]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank6]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank6]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank6]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank6]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank6]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank6]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank6]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank6]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank6]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank6]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank6]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank6]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank6]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank6]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank6]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank6]: frame #41: + 0x29d90 (0x7efda74fed90 in /lib/x86_64-linux-gnu/libc.so.6) [rank6]: frame #42: __libc_start_main + 0x80 (0x7efda74fee40 in /lib/x86_64-linux-gnu/libc.so.6) [rank6]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank6]: . This may indicate a possible application crash on rank 0 or a network set up issue. srun: error: h100-ord01-03-592: task 6: Exited with exit code 1 [rank1]:[W1114 23:18:31.548381325 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 1] using GPU 1 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank1]: Traceback (most recent call last): [rank1]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank1]: main() [rank1]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank1]: dist.barrier() [rank1]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank1]: return func(*args, **kwargs) [rank1]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank1]: work = group.barrier(opts=opts) [rank1]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank1]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank1]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f86ebcb8cf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank1]: frame #1: + 0x6021e58 (0x7f86db8e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank1]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f86db8debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank1]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f86db8def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank1]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f86db8dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank1]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f86db88ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank1]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f86db88ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank1]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f86db88ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank1]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f868e55adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank1]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f868e5674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank1]: frame #10: + 0x1205255 (0x7f868e56d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank1]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f868e56e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank1]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f868e57f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank1]: frame #13: + 0x5fc28a2 (0x7f86db8828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank1]: frame #14: + 0x5fcd035 (0x7f86db88d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank1]: frame #15: + 0x55d8ceb (0x7f86dae98ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank1]: frame #16: + 0x55d6579 (0x7f86dae96579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank1]: frame #17: + 0x1aa7828 (0x7f86d7367828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank1]: frame #18: + 0x5fd6d74 (0x7f86db896d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank1]: frame #19: + 0x5fd7b05 (0x7f86db897b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank1]: frame #20: + 0xe1df28 (0x7f86eb383f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank1]: frame #21: + 0x4cf144 (0x7f86eaa35144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank1]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank1]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank1]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank1]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank1]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank1]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank1]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank1]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank1]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank1]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank1]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank1]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank1]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank1]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank1]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank1]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank1]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank1]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank1]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank1]: frame #41: + 0x29d90 (0x7f8701c9ad90 in /lib/x86_64-linux-gnu/libc.so.6) [rank1]: frame #42: __libc_start_main + 0x80 (0x7f8701c9ae40 in /lib/x86_64-linux-gnu/libc.so.6) [rank1]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank1]: . This may indicate a possible application crash on rank 0 or a network set up issue. srun: error: h100-ord01-03-592: task 1: Exited with exit code 1 [rank3]:[W1114 23:18:45.963407797 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 3] using GPU 3 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank3]: Traceback (most recent call last): [rank3]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank3]: main() [rank3]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank3]: dist.barrier() [rank3]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank3]: return func(*args, **kwargs) [rank3]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank3]: work = group.barrier(opts=opts) [rank3]: torch.distributed.DistBackendError: [3] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank3]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank3]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f379fb6bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank3]: frame #1: + 0x6021e58 (0x7f378f6e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank3]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f378f6debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank3]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f378f6def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank3]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f378f6dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank3]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f378f68ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank3]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f378f68ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank3]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f378f68ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank3]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f374235adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank3]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f37423674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank3]: frame #10: + 0x1205255 (0x7f374236d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank3]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f374236e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank3]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f374237f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank3]: frame #13: + 0x5fc28a2 (0x7f378f6828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank3]: frame #14: + 0x5fcd035 (0x7f378f68d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank3]: frame #15: + 0x55d8ceb (0x7f378ec98ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank3]: frame #16: + 0x55d6579 (0x7f378ec96579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank3]: frame #17: + 0x1aa7828 (0x7f378b167828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank3]: frame #18: + 0x5fd6d74 (0x7f378f696d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank3]: frame #19: + 0x5fd7b05 (0x7f378f697b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank3]: frame #20: + 0xe1df28 (0x7f379f183f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank3]: frame #21: + 0x4cf144 (0x7f379e835144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank3]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank3]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank3]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank3]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank3]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank3]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank3]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank3]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank3]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank3]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank3]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank3]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank3]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank3]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank3]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank3]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank3]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank3]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank3]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank3]: frame #41: + 0x29d90 (0x7f37b5c1ad90 in /lib/x86_64-linux-gnu/libc.so.6) [rank3]: frame #42: __libc_start_main + 0x80 (0x7f37b5c1ae40 in /lib/x86_64-linux-gnu/libc.so.6) [rank3]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank3]: . This may indicate a possible application crash on rank 0 or a network set up issue. srun: error: h100-ord01-03-592: task 3: Exited with exit code 1 [rank2]:[W1114 23:18:49.802816107 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 2] using GPU 2 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank2]: Traceback (most recent call last): [rank2]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank2]: main() [rank2]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank2]: dist.barrier() [rank2]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank2]: return func(*args, **kwargs) [rank2]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank2]: work = group.barrier(opts=opts) [rank2]: torch.distributed.DistBackendError: [2] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank2]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank2]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f263056bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank2]: frame #1: + 0x6021e58 (0x7f26200e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank2]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f26200debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank2]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f26200def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank2]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f26200dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank2]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f262008ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank2]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f262008ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank2]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f262008ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank2]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f25d2d5adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank2]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f25d2d674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank2]: frame #10: + 0x1205255 (0x7f25d2d6d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank2]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f25d2d6e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank2]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f25d2d7f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank2]: frame #13: + 0x5fc28a2 (0x7f26200828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank2]: frame #14: + 0x5fcd035 (0x7f262008d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank2]: frame #15: + 0x55d8ceb (0x7f261f698ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank2]: frame #16: + 0x55d6579 (0x7f261f696579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank2]: frame #17: + 0x1aa7828 (0x7f261bb67828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank2]: frame #18: + 0x5fd6d74 (0x7f2620096d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank2]: frame #19: + 0x5fd7b05 (0x7f2620097b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank2]: frame #20: + 0xe1df28 (0x7f262fb83f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank2]: frame #21: + 0x4cf144 (0x7f262f235144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank2]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank2]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank2]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank2]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank2]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank2]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank2]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank2]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank2]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank2]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank2]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank2]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank2]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank2]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank2]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank2]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank2]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank2]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank2]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank2]: frame #41: + 0x29d90 (0x7f26465aed90 in /lib/x86_64-linux-gnu/libc.so.6) [rank2]: frame #42: __libc_start_main + 0x80 (0x7f26465aee40 in /lib/x86_64-linux-gnu/libc.so.6) [rank2]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank2]: . This may indicate a possible application crash on rank 0 or a network set up issue. srun: error: h100-ord01-03-592: task 2: Exited with exit code 1