Faster LLMs: Accelerate Inference with Speculative Decoding

Transcript

So you want your large language model to be fast. Let me show you how. Speculative decoding is an effective technique for speeding up LLM inference times without sacrificing quality of output. This approach follows the slogan draft and verify by using a smaller draft model to speculate about future tokens while a larger target model verifies them i... Read More