vLLM

An open-source engine for serving large language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code., designed to fit as many simultaneous requests as possible onto each GPU. Its founding idea, PagedAttentionPagedAttention, prop. n.: A way of storing an AI model's working memory in small fixed-size blocks, like the pages of a computer's memory, so little space is wasted and more users can be served at once., stores the KV cacheKV cache, n.: A memory store where a chatbot keeps its working notes on everything already read and written, so each new word does not require re-reading the whole conversation from scratch. in fixed-size blocks allocated on demand, much as an operating system pages memory, so that little GPU memory is wasted. Around that core it schedules work with continuous batchingcontinuous batching, n.: A way for an AI server to handle many conversations together, letting new ones join and finished ones leave after every word, instead of waiting for a whole group to finish. and chunked prefillchunked prefill, n.: A serving trick that splits the reading of a long prompt into smaller pieces and slots them between other users' replies, so nobody's answer freezes while a huge document is read., reuses shared prompt prefixes (prompt cachingprompt caching, n.: Saving the work a model did reading the start of a prompt, so later requests that begin the same way can skip that reading and come back faster and cheaper.), speeds generation with quantizationquantization, n.: Storing an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy. (formats such as GPTQGPTQ, prop. n.: A popular method for shrinking large language models to about 4 bits per number after training, carefully adjusting the remaining numbers to make up for each rounding error., AWQAWQ, abbr.: A method for shrinking large language models to about 4 bits per number that protects the small share of numbers most important to the model, judged by the data flowing through them. and FP8FP8, abbr.: An 8-bit number format that uses a quarter of the memory of the standard 32-bit format, letting new AI chips train and run models roughly twice as fast as with 16 bits.) and speculative decodingspeculative decoding, n.: A speed-up trick in which a small, fast model guesses the next few words and the big model checks all the guesses at once, keeping the right ones: faster answers, same quality., spreads large models across devices with tensortensor parallelism, n.: Splitting the individual giant tables of numbers inside each layer of a model across several chips, so that every chip does a slice of every calculation at the same time., pipelinepipeline parallelism, n.: Splitting a model's layers into consecutive stages on different chips, like stations on an assembly line, and streaming small batches of data through so that every station stays busy. and expert parallelismexpert parallelism, n.: A way of running a 'mixture of experts' model — one made of many specialist sub-networks — by giving each chip its own set of specialists and sending each piece of text to whichever chip holds the specialists it needs., and answers clients through an OpenAI-compatible APIOpenAI-compatible API, n.: An AI service that accepts the same request format as OpenAI's, so software written for OpenAI's models can use other companies' or self-hosted models by changing an address and key.. Built on PyTorchPyTorch, prop. n.: The most widely used free software toolkit for building and training neural networks, especially in research; programmers write ordinary Python and it handles the calculus and the graphics chips., it supports more than a hundred model architectures and runs on GPUs from NVIDIA, AMD and Intel as well as CPUs, with plugins for TPUsTPU, abbr.: Google's own family of chips built specifically for AI, used to train and run its Gemini models and rented to other companies through its cloud. and other accelerators. It is a standard baseline in serving research and the engine inside many commercial inference servicesinference provider, n.: A company that runs AI models on its own computers and sells access to them over the internet, charging by use, so customers need not buy or operate the hardware themselves..

We serve the fine-tuned model with vLLM behind an OpenAI-compatible endpoint.