Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention

Authors: Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Y.X. Wei, Lean Wang, Zhiping Xiao, Yuqing Wang, Chong Ruan, Ming Zhang, Wenfeng Liang, Wangding Zeng

Year: 2025

Venue: arXiv

Type: article

URL: https://arxiv.org/abs/2502.11089

arXiv: 2502.11089

Cite as: [@yuan2025native_sparse_attention]

Raw Files

No raw files yet. Run node scripts/fetch-bibliography-raw.mjs --only yuan2025native_sparse_attention to populate, or drop files into raw/bibliography/yuan2025native_sparse_attention/.

BibTeX

@article{yuan2025native_sparse_attention,
  title = {Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention},
  author = {Jingyang Yuan and Huazuo Gao and Damai Dai and Junyu Luo and Liang Zhao and Zhengyan Zhang and Zhenda Xie and Y.X. Wei and Lean Wang and Zhiping Xiao and Yuqing Wang and Chong Ruan and Ming Zhang and Wenfeng Liang and Wangding Zeng},
  year = {2025},
  journal = {arXiv},
  url = {https://arxiv.org/abs/2502.11089}
}

Notes

No notes yet. Create notes/yuan2025native_sparse_attention.md to add notes.