Fast Linear Algebra on GPU

GPU; parallel reduction; linear algebra; BLAS; OpenCL; CUDA

GPUs have been successfully used for acceleration of many mathematical functions and libraries. A common limitation of those libraries is the minimal size of primitives being handled, in order to achieve a significant speedup compared to their CPU versions. The minimal size requirement can prove prohibitive for many applications. It can be loosened by batching operations in order to have sufficient amount of data to perform the calculation maximally efficiently on the GPU. A fast OpenCL implementation of two basic vector functions - vector reduction and vector scaling - is described in this paper. Its performance is analyzed by running benchmarks on two of the most common GPUs in use - Tesla and Fermi GPUs from NVIDIA. Reported experimental results show that our implementation significantly outperforms the current state-of-the-art GPU-based basic linear algebra library CUBLAS.