DeepSeek/DeepEP
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepEP synced 2025-06-26 18:28:11 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

In the Internode Normal Kernel, when using nvshmem ibrc for RDMA data transmission, a single QP is used for data transfer between two GPUs, which limits kernel performance in network card dual-port and RoCE network scenarios.

Browse Source 
In our optimized Internode Normal Kernel, we implemented multiple QPs for data transmission between two GPUs, setting a different QP for each channel. Additionally, we modified the transmission method from IBRC to IBGAD.

Through these optimizations, the Internode Normal Kernel achieves optimal performance in both H800 and H20 environments, with RDMA transmission performance nearly reaching the physical network performance limit. Using the current default statistical method, in 4-node H800 and H20 environments, RDMA performance can reach 60GB/s+.
This commit is contained in:
moningchen
2025-04-21 15:37:19 +08:00
parent a84a24808f
commit 5ab80c28f3
5 changed files with 414 additions and 23 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
csrc/kernels/runtime.cu
View File

@@ -59,7 +59,8 @@ int init(const std::vector<uint8_t> &root_unique_id_val, int rank, int num_ranks
}
// Normal operations use IBRC, while low-latency operations use IBGDA
if (low_latency_mode) {
bool internode_use_ibgda = true;
if (low_latency_mode or internode_use_ibgda) {
nvshmemi_device_host_state_t* dev_state_ptr = nullptr;
CUDA_CHECK(cudaGetSymbolAddress(reinterpret_cast<void**>(&dev_state_ptr), nvshmemi_device_state_d));