Zhean Xu
04278f6dee
Weight gradient kernels for dense and MoE models ( #95 )
...
* Init weight gradient kernels.
* Support unaligned n,k and gmem stride
* Update docs
* Several cleanups
* Remove restrictions on N
* Add stride(0) assertions
---------
Co-authored-by: Chenggang Zhao <chenggangz@deepseek.com >
2025-05-14 14:47:58 +08:00
Gabriel Wu
bfe983c4c2
Refactor JIT compilation (+NVRTC support) ( #94 )
...
* [wip] refactor: compile to .cubin
Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com >
* refactor: compile to .cubin and add NVRTC option
Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com >
* fix: compiler version
Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com >
* feat: compat for old drivers
Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com >
* feat: save kernel name to file
Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com >
* feat: fix win compat
Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com >
* fix: windows compat
Signed-off-by: Gabriel Wu <13583761+lucifer1004@users.noreply.github.com >
* feat: make API more general
Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com >
* feat: drop support for CUDA<12.3
Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com >
* doc: update README
Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com >
* Some lints and refactor
* Refactor runtime
* Several fixes
* Refactor environment variables
* Code format
* Add a TODO
* Compatible with CUDA 12.3
* Fix indent
* Fix typing
* Drop support for Windows
* Add a TODO
---------
Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com >
Signed-off-by: Gabriel Wu <13583761+lucifer1004@users.noreply.github.com >
Co-authored-by: Chenggang Zhao <chenggangz@deepseek.com >
2025-05-07 11:38:14 +08:00
ademeure
6cbff5778f
Correctly flush L2, as reconstructing the tensors on every iteration effectively put them in the L2, and gave the GPU enough idle time to avoid thermal throttling in a potentially unrealistic way.
...
The previous behaviour is potentially representative of some use cases (e.g. previous kernel filling L2 with the data in a very specific way) but not standard benchmarking practice.
2025-03-15 20:46:24 +00:00
Chenggang Zhao
a6d97a1c1b
Initial commit
2025-02-25 22:52:41 +08:00
