llama.cpp, NVIDIA Improve ggml Multi-GPU Performance on RTX Systems
llama.cpp maintainers and NVIDIA engineers have spent the past few months improving multi-GPU performance in ggml, resulting in significant gains on RTX systems and laying the groundwork for hardware-agnostic tensor parallelism. The work was highlighted as part of recent advances in the low-level inference stack used by llama.cpp.
The changes were presented alongside new NVIDIA and Microsoft tools for Windows PCs aimed at on-device personal AI agents, including secure sandboxing, faster local inference, multi-GPU support and RTX acceleration for Windows AI APIs. That adds context for how the performance work could support broader local AI deployment on PCs.
From the sources (1 posts)
@ggerganovHighlighting recent advances in multi-GPU and tensor parallel support in llama.cpp Over the last few months llama.cpp maintainers and engineers from NVIDIA collaborated to improve the multi-GPU performance in ggml. This resulted in signif