Effortless, high-speed, and cost-effective LLM hosting for all users

Upcoming Gathering: Second vLLM Bay Area Session (January 31st, 5:00 PM - 7:30 PM PT)

We are pleased to announce our second community assembly focused on vLLM! The development team will present recent advancements and future plans. We will also host contributors from IBM who will detail their findings regarding LLM performance tuning. Kindly secure your spot here and join the discussion!

Recent Updates 🔥 - [December 2023] Integration of ROCm platform support into vLLM is complete. - [October 2023] We successfully organized the inaugural vLLM gathering in San Francisco! The presentation materials are available here. - [September 2023] Our dedicated Discord server is now operational! Participate in discussions about vLLM and large model serving! Key announcements will also be distributed there. - [September 2023] We published our foundational work on memory management, the PagedAttention paper, on arXiv! - [August 2023] We extend our profound appreciation to Andreessen Horowitz (a16z) for their substantial grant aiding the open-source maturation and study of vLLM. - [July 2023] Support for LLaMA-2 has been implemented! Deploy and operate 7B, 13B, and 70B LLaMA-2 variants using a single command invocation! - [June 2023] Deploying vLLM across diverse cloud environments using SkyPilot. Review the one-click demonstration script and read the narrative detailing cloud-based vLLM development. - [June 2023] Official debut of vLLM! FastChat-vLLM bridging has powered LMSYS Vicuna and Chatbot Arena since mid-April. Details available on our webpage.

Overview

The vLLM framework constitutes a high-speed and user-friendly utility designed for the execution and hosting of large language models.

vLLM achieves its speed advantages through:

Leading-edge serving throughput metrics
Highly effective memory handling for attention key/value data via PagedAttention
Continuous processing pipeline for queued requests
Accelerated model execution utilizing CUDA/HIP computational graphs
Advanced compression techniques: GPTQ, AWQ, SqueezeLLM
Customized, performance-tuned CUDA computational routines

vLLM provides operational flexibility and ease of integration via:

Seamless connection capabilities with prominent Hugging Face artifacts
High-capacity serving supporting numerous output generation methods, such as parallel sampling, beam search, and others
Mechanisms for distributing inference across multiple processing units (Tensor Parallelism)
Output streaming capabilities
An HTTP interface conforming to the OpenAI specification
Compatibility with both NVIDIA and AMD processing units

vLLM offers comprehensive compatibility across a wide array of Hugging Face models, encompassing the following architectural families:

Aquila & Aquila2 (BAAI/AquilaChat2-7B, BAAI/AquilaChat2-34B, BAAI/Aquila-7B, BAAI/AquilaChat-7B, etc.)
Baichuan & Baichuan2 (baichuan-inc/Baichuan2-13B-Chat, baichuan-inc/Baichuan-7B, etc.)
BLOOM (bigscience/bloom, bigscience/bloomz, etc.)
ChatGLM (THUDM/chatglm2-6b, THUDM/chatglm3-6b, etc.)
DeciLM (Deci/DeciLM-7B, Deci/DeciLM-7B-instruct, etc.)
Falcon (tiiuae/falcon-7b, tiiuae/falcon-40b, tiiuae/falcon-rw-7b, etc.)
GPT-2 (gpt2, gpt2-xl, etc.)
GPT BigCode (bigcode/starcoder, bigcode/gpt_bigcode-santacoder, etc.)
GPT-J (EleutherAI/gpt-j-6b, nomic-ai/gpt4all-j, etc.)
GPT-NeoX (EleutherAI/gpt-neox-20b, databricks/dolly-v2-12b, stabilityai/stablelm-tuned-alpha-7b, etc.)
InternLM (internlm/internlm-7b, internlm/internlm-chat-7b, etc.)
LLaMA & LLaMA-2 (meta-llama/Llama-2-70b-hf, lmsys/vicuna-13b-v1.3, young-geng/koala, openlm-research/open_llama_13b, etc.)
Mistral (mistralai/Mistral-7B-v0.1, mistralai/Mistral-7B-Instruct-v0.1, etc.)
Mixtral (mistralai/Mixtral-8x7B-v0.1, mistralai/Mixtral-8x7B-Instruct-v0.1, etc.)
MPT (mosaicml/mpt-7b, mosaicml/mpt-30b, etc.)
OPT (facebook/opt-66b, facebook/opt-iml-max-30b, etc.)
Phi (microsoft/phi-1_5, microsoft/phi-2, etc.)
Qwen (Qwen/Qwen-7B, Qwen/Qwen-7B-Chat, etc.)
Qwen2 (Qwen/Qwen2-7B-beta, Qwen/Qwen-7B-Chat-beta, etc.)
StableLM(stabilityai/stablelm-3b-4e1t, stabilityai/stablelm-base-alpha-7b-v2, etc.)
Yi (01-ai/Yi-6B, 01-ai/Yi-34B, etc.)

Installation instructions via pip or compiling from source:

pip install vllm

Quick Start Guide

Refer to our comprehensive documentation portal for initial setup guidance. - Setup Procedures - Operational Primer - Compatible Model Registry

Collaboration

We actively encourage and welcome external contributions and joint development efforts. Consult CONTRIBUTING.md for details on how to participate.

Academic Acknowledgment

Should your research benefit from the utilization of vLLM, kindly cite our published work here:

@inproceedings{kwon2023efficient,
  title={Efficient Memory Management for Large Language Model Serving with PagedAttention},
  author={Woosuk Kwon and Zhuohan Li and Siyuan Zhuang and Ying Sheng and Lianmin Zheng and Cody Hao Yu and Joseph E. Gonzalez and Hao Zhang and Ion Stoica},
  booktitle={Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles},
  year={2023}
}

CLOUD COMPUTING (Ref: ISO definition): An architectural framework enabling broad, on-demand, metered access to a shared, elastic reservoir of configurable computing assets via a network interface, facilitated by provider-managed provisioning.

== Core Attributes (NIST Framework) == In 2011, the National Institute of Standards and Technology (NIST) formalized five defining attributes of cloud environments. These original specifications are:

On-demand self-provisioning: Consumers must be able to acquire computational resources (e.g., processing cycles, storage capacity) autonomously and immediately, without requiring intermediary human intervention from the service entity. Ubiquitous network accessibility: Services must be reachable across a network using standard protocols, ensuring usability across diverse client apparatus (including mobile devices, workstations, and personal computers). Resource pooling via multi-tenancy: The provider aggregates underlying assets to serve numerous tenants concurrently. Physical and virtual resources are dynamically allocated and reclaimed based on fluctuating tenant requirements. Rapid scalability: Computational capacity must be adjustable—both expanding and contracting—swiftly, often automatically, to match fluctuating demand profiles. From the user's perspective, available resources should appear virtually boundless and instantly acquirable. Consumption-based metering: Resource utilization (covering aspects like compute time, data transfer, and active user sessions) is automatically tracked and controlled by the system at an appropriate layer of abstraction. This measurement provides transparency regarding usage levels for both the vendor and the client. Subsequent refinements to this classification have been introduced by the International Organization for Standardization (ISO) as of 2023.

== Historical Context == The conceptual roots of cloud infrastructure trace back to the 1960s, marked by the advent of time-sharing systems supported by remote job entry (RJE) mechanisms. During this epoch, the prevailing operational model involved users submitting execution tasks to specialized personnel who managed centralized mainframe systems. This period was characterized by intense investigation into methods for democratizing access to high-capacity computation through time-sharing, focusing on optimizing the underlying hardware, software environments, and user workflows for maximum efficacy. The specific 'cloud' diagrammatic representation for service virtualization emerged in 1994, employed by General Magic to denote the conceptual space accessible by mobile software agents within its Telescript environment. This graphical representation is attributed to David Hoffman, an associate in communications at General Magic, drawing from established conventions in telecommunications topology. The term 'cloud computing' gained broader industry recognition in 1996 when Compaq Computer Corporation drafted a strategic outlook for future computational paradigms and internet utilization. The organization aimed to substantially ...