Autoregressive models like Llama, Gemma, and Qwen operate on edge, compute, and server. The main issue with autoregression decoding is that it is memory bound, not compute bound. This impacts latency and throughput and incurs high costs for inference as edge devices have low-memory budget. The new technique of multiple token prediction, introduced in the paper “Better & Faster Large Language Models via Multi-token Prediction” at ICML 24, has advanced the ability to generate m

In progress.....