As much as 40x quicker than llama.cpp?

Operating giant language fashions (LLMs) on shopper {hardware} may be difficult. If the LLM doesn’t match on the GPU reminiscence, quantization is often utilized to cut back its dimension. Nevertheless, even after quantization, the mannequin may nonetheless be too giant to suit on the GPU. Another is to run it on the CPU RAM utilizing a framework optimized for CPU inference similar to llama.cpp.

Intel can also be engaged on accelerating inference on the CPU. They suggest a framework, Intel’s extension for Transformers, constructed on high of Hugging Face Transformers and straightforward to make use of to use the CPU.

With Neural Pace (Apache 2.0 license), which depends on Intel’s extension for Transformers, Intel additional accelerates inference for 4-bit LLMs on CPUs. In line with Intel, utilizing this framework can make inference as much as 40x quicker than llama.cpp.

On this article, I overview the primary optimizations Neural Pace brings. I present easy methods to use it and benchmark the inference throughput. I additionally evaluate it with llama.cpp.

At NeurIPS 2023, Intel introduced the primary optimizations for inference on CPUs:

Environment friendly LLM Inference on CPUs

Within the following determine, the parts in inexperienced are the primary additions introduced by Neural Pace for environment friendly inference:

The CPU tensor library supplies a number of kernels optimized for inference with 4-bit fashions. They assist x86 CPUs, together with AMD CPUs.

These kernels are optimized for fashions quantized with the INT4 knowledge sort. GPTQ, AWQ, and GGUF fashions are supported and accelerated by Neural Pace. Furthermore, Intel additionally has its personal quantization library, Neural Compressor, that shall be known as if the mannequin is just not quantized.

As for the “LLM Optimizations” highlighted within the determine above, Intel doesn’t write a lot about them within the NeurIPS paper.