Chenggang Zhao
185ecf5c4a
Merge remote-tracking branch 'origin/main' into internode-tma
...
# Conflicts:
# csrc/kernels/configs.cuh
# csrc/kernels/internode.cu
2025-06-24 09:29:07 +08:00
Chenggang Zhao
a15faa9ff0
Remove useless assertion
2025-06-24 09:21:35 +08:00
Chenggang Zhao
bc118b248a
Add the transaction window data structure for RDMA senders ( #245 )
...
* Add draft
* Add fast-debugging flags
* Fix several bugs
* Add sender timeout checks
* Fix stuck
* Fix bugs
* Fix bugs
2025-06-24 09:12:40 +08:00
Shangyan Zhou
9eb2f84b3e
Optimize intranode combine. ( #247 )
...
* Increase the test round.
* Add warp synchronization.
* Shuffle the send warps.
* Add time elapsed into bench result.
2025-06-24 09:10:23 +08:00
fzyzcjy
fbcf430006
Update internode_ll.cu ( #246 )
2025-06-23 15:18:10 +08:00
fzyzcjy
c95997f8c4
Update deep_ep.cpp ( #242 )
2025-06-23 11:44:06 +08:00
Chenggang Zhao
7b0c25f864
Support more hidden size
2025-06-20 16:37:28 +08:00
Chenggang Zhao
a086ac5536
Use correct buffer pointers
2025-06-20 16:25:49 +08:00
Chenggang Zhao
782b40a8ff
Add ENABLE_FAST_DEBUG
2025-06-20 14:44:53 +08:00
Chenggang Zhao
47dd77ab5f
Add retired flag
2025-06-20 14:35:15 +08:00
Chenggang Zhao
74afd75df2
Fix bugs
2025-06-20 14:27:54 +08:00
Chenggang Zhao
371df2da52
Fix bugs
2025-06-20 13:44:49 +08:00
Chenggang Zhao
8da790e3f3
Fix the shifted buffer pointer
2025-06-20 11:31:57 +08:00
Chenggang Zhao
cd5c57fb2a
Fix compilation
2025-06-20 11:15:03 +08:00
Chenggang Zhao
49b9084268
Fix several bugs
2025-06-20 10:57:56 +08:00
Chenggang Zhao
177e491e92
Fix send heads
2025-06-19 18:05:59 +08:00
Chenggang Zhao
55bbd8caaf
Add impl
2025-06-19 17:15:43 +08:00
Chenggang Zhao
a0a6e22eff
Fully remove forwarders' and NVL receivers' code
2025-06-19 13:48:07 +08:00
Chenggang Zhao
3a3398f686
Minor fix
2025-06-19 10:38:42 +08:00
Shangyan Zhou
77f97f79bd
Fix the tail loading issue. ( #219 )
...
* Fix the tail loading issue.
* Modify the sync offset.
2025-06-18 09:23:25 +08:00
Shangyan Zhou
dd133d39bc
Fix warp synchronization. ( #215 )
...
* Fix warp synchronization.
* Another fix.
2025-06-16 17:05:11 +08:00
Chenggang Zhao
8aaddf76ae
Remove the low-latency usage flag ( #214 )
2025-06-16 13:30:14 +08:00
Chenggang Zhao
1b92be8a71
Add automatic warp count control for low-latency kernels ( #213 )
...
* Add automatic warp count control for low-latency dispatch
* Add automatic warp count control for low-latency combine
* More assertions
2025-06-16 11:56:43 +08:00
fzyzcjy
4e923188f7
Update intranode.cu ( #210 )
2025-06-16 11:03:58 +08:00
Shangyan Zhou
483f00af84
Update assertion of num_rc_per_pe.
2025-06-13 15:16:23 +08:00
Zhicheng Wu
05df5554ff
Use one qp per sm for internode normal kernels ( #181 )
...
let the sender SM use the channel_id, and the receiver SM use channel_id + num_channels
2025-06-13 14:37:59 +08:00
Shifang Xu
21efbe9b48
Support UE8M0 data format. ( #206 )
2025-06-12 09:38:19 +08:00
Chenggang Zhao
b8d90fb753
Support Ampere architecture ( #204 )
...
* Update README
* Update `setup.py`
* Fix headers
* Add `DISABLE_NVSHMEM` for APIs
* Fix launch
* Fix TMA settings
* Fix TMA usages
* Fix dlink
* Separate layout kernels
* Update version
* Add `is_sm90_compiled`
* Fix tests
* Add NVLink connection checks
* Update README
* Fix tests
* Add some comments
* Minor fix
* Minor fix
* Fix bugs
2025-06-11 15:48:18 +08:00
Chenggang Zhao
a8299ca7c2
Support CUDA graph for intranode normal kernels ( #203 )
2025-06-11 11:08:54 +08:00
Chenggang Zhao
8da2d7b38d
Fully remove barrier FIFO designs ( #200 )
...
* Fully remove FIFO slots
* Fully remove FIFO buffers
* Minor fix styles
* Fix some typos
* Bugs fixed
* Cleanup `ibgda_poll_cq`
2025-06-10 16:23:20 +08:00
Chenggang Zhao
1157693c0c
Remove useless comments
2025-06-09 17:14:25 +08:00
Chenggang Zhao
5a2e37fa28
Support statistics tensor for low-latency kernels ( #196 )
2025-06-09 15:50:56 +08:00
Chenggang Zhao
0d1a855d81
Add low-latency kernel PCIe usage flag ( #195 )
...
* Add low-latency kernel usage flag
* Update comments
2025-06-09 14:37:13 +08:00
Chenggang Zhao
564e375234
Fix < PTX ISA 8.6 compatibility ( #194 )
2025-06-09 10:48:42 +08:00
Chenggang Zhao
c8dceba110
Use TMA instead of LD/ST for intra-node normal kernels ( #191 )
...
* Update CMake files
* Use TMA instead of LD/ST for intranode dispatch
* Use TMA instead of LD/ST for intranode combine
* Adjust configs
* Test default configs as well
* More warps for combine
* Add inter-thread fence
* Enable more warps
* Do not use TMA for senders
* Update configs
* Remove useless wait
2025-06-06 15:40:17 +08:00
wzc.wuzhicheng
d0225df27d
Fix notify_dispatch: using warp 0 to issue send
...
Signed-off-by: wzc.wuzhicheng <wzc.wuzhicheng@linux.alibaba.com >
2025-06-03 20:20:02 +08:00
Shangyan Zhou
9fe9021f29
Use IBGDA only ( #177 )
2025-05-28 16:40:14 +08:00
Chenggang Zhao
92405ddf30
Code cleanup and bug fixed
2025-05-23 11:14:16 +08:00
cywork121
68ae8b3d07
Feature: LL nvlink p2p ( #173 )
2025-05-23 10:37:45 +08:00
sleepcoo
a107266a4e
support hidden size 4096
...
Co-authored-by: zhyncs <me@zhyncs.com >
Co-authored-by: yinfan98 <1106310035@qq.com >
2025-05-12 16:41:21 +08:00
wangfakang
63c29d06a0
To mitigate incast congestion, shuffle the starting index of target rank for different ranks and channels
...
Signed-off-by: wangfakang <fakangwang@gmail.com >
2025-05-10 09:55:35 +08:00
fzyzcjy
adc6e24cb0
Update deep_ep.cpp
2025-05-08 16:01:47 +08:00
fzyzcjy
23ded3bd8d
Update deep_ep.cpp
2025-04-29 09:58:31 +08:00
Shangyan Zhou
e255d57bef
Use put_nbi_warp.
2025-04-22 12:29:46 +08:00
Chenggang Zhao
edbb1bc3ff
Several code lints
2025-04-22 10:52:10 +08:00
Shangyan Zhou
3e54b78fd7
Normal kernels always use IBGDA mode.
2025-04-22 10:36:24 +08:00
Shangyan Zhou
20b2aaaf9e
Refactor some code.
2025-04-22 10:22:30 +08:00
Shangyan Zhou
e2c578485c
Revert ibgda_device.cuh and remove some comments.
2025-04-21 17:44:32 +08:00
moningchen
5ab80c28f3
In the Internode Normal Kernel, when using nvshmem ibrc for RDMA data transmission, a single QP is used for data transfer between two GPUs, which limits kernel performance in network card dual-port and RoCE network scenarios.
...
In our optimized Internode Normal Kernel, we implemented multiple QPs for data transmission between two GPUs, setting a different QP for each channel. Additionally, we modified the transmission method from IBRC to IBGAD.
Through these optimizations, the Internode Normal Kernel achieves optimal performance in both H800 and H20 environments, with RDMA transmission performance nearly reaching the physical network performance limit. Using the current default statistical method, in 4-node H800 and H20 environments, RDMA performance can reach 60GB/s+.
2025-04-21 15:50:39 +08:00
Chenggang Zhao
42494864ba
Remove useless control metadata for low-latency combine
2025-04-07 09:55:39 +08:00
