Skip to content

vllm.v1.core.kv_cache_config_builder

Interface and resolution for pluggable KV cache config builders.

Classes:

Functions:

KVCacheConfigBuilder

Bases: ABC

Interface for platform KV cache placement.

.. note:: This interface is experimental and subject to change. The exact hooks and the num_blocks contract may evolve before it stabilizes.

Subclasses normally inherit from DefaultKVCacheConfigBuilder and override only the hooks they need. The cross-worker planning flow (:func:vllm.v1.core.kv_cache_planning.get_kv_cache_configs) is Core-owned; this builder supplies the placement decisions: how layers are split into groups and how a group list is materialized into tensors for a given block count. Grouping coordination lives in DefaultKVCacheConfigBuilder (a model implementing :class:~vllm.model_executor.models.interfaces. SupportsKVCacheGrouping owns its grouping there); a platform that replaces the builder fully owns its own grouping.

Methods:

Source code in vllm/v1/core/kv_cache_config_builder.py 
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
class KVCacheConfigBuilder(ABC):
    """Interface for platform KV cache placement.

    .. note::
        This interface is experimental and subject to change. The exact hooks
        and the ``num_blocks`` contract may evolve before it stabilizes.

    Subclasses normally inherit from ``DefaultKVCacheConfigBuilder`` and
    override only the hooks they need. The cross-worker planning flow
    (:func:`vllm.v1.core.kv_cache_planning.get_kv_cache_configs`) is Core-owned;
    this builder supplies the placement decisions: how layers are split into
    groups and how a group list is materialized into tensors for a given block
    count. Grouping coordination lives in ``DefaultKVCacheConfigBuilder`` (a
    model implementing :class:`~vllm.model_executor.models.interfaces.
    SupportsKVCacheGrouping` owns its grouping there); a platform that replaces
    the builder fully owns its own grouping.
    """

    @abstractmethod
    def get_kv_cache_groups(
        self,
        vllm_config: "VllmConfig",
        kv_cache_spec: dict[str, "KVCacheSpec"],
    ) -> list["KVCacheGroupSpec"]:
        """Split the given layers into KV cache groups.

        Planning passes all workers' layers; profiling passes one worker's.
        Core rejects a result that does not put each layer in exactly one group.

        The default builder honors a model's
        :class:`~vllm.model_executor.models.interfaces.SupportsKVCacheGrouping`
        declaration here before falling back to the default grouping; a
        platform-specific builder decides its own grouping.
        """
        raise NotImplementedError

    @abstractmethod
    def get_kv_cache_config_from_groups(
        self,
        vllm_config: "VllmConfig",
        kv_cache_groups: list["KVCacheGroupSpec"],
        num_blocks: int,
    ) -> "KVCacheConfig":
        """Materialize the groups for exactly ``num_blocks`` blocks.

        Each device backing's size is ``bytes_per_block * num_blocks``. All
        device tensors share one backing allocation unless the builder
        overrides ``_get_pool_bytes_per_block``. Host-resident tensors, such as
        HiSparse source storage, are excluded from the device byte accounting
        and sized independently.
        """
        raise NotImplementedError

    def _get_pool_bytes_per_block(
        self,
        vllm_config: "VllmConfig",
        kv_cache_groups: list["KVCacheGroupSpec"],
    ) -> int | None:
        """Return the device bytes one block ID consumes, or ``None``.

        Private and transitional; expected to go away with worker-driven
        allocation (RFC #42449). With ``None`` (the default), Core derives the
        value from a one-block placement with one device backing. Override it
        only for placements with several device allocations.
        """
        return None

_get_pool_bytes_per_block(vllm_config, kv_cache_groups)

Return the device bytes one block ID consumes, or None.

Private and transitional; expected to go away with worker-driven allocation (RFC #42449). With None (the default), Core derives the value from a one-block placement with one device backing. Override it only for placements with several device allocations.

Source code in vllm/v1/core/kv_cache_config_builder.py 
72
73
74
75
76
77
78
79
80
81
82
83
84
def _get_pool_bytes_per_block(
    self,
    vllm_config: "VllmConfig",
    kv_cache_groups: list["KVCacheGroupSpec"],
) -> int | None:
    """Return the device bytes one block ID consumes, or ``None``.

    Private and transitional; expected to go away with worker-driven
    allocation (RFC #42449). With ``None`` (the default), Core derives the
    value from a one-block placement with one device backing. Override it
    only for placements with several device allocations.
    """
    return None

get_kv_cache_config_from_groups(vllm_config, kv_cache_groups, num_blocks) abstractmethod

Materialize the groups for exactly num_blocks blocks.

Each device backing's size is bytes_per_block * num_blocks. All device tensors share one backing allocation unless the builder overrides _get_pool_bytes_per_block. Host-resident tensors, such as HiSparse source storage, are excluded from the device byte accounting and sized independently.

Source code in vllm/v1/core/kv_cache_config_builder.py 
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
@abstractmethod
def get_kv_cache_config_from_groups(
    self,
    vllm_config: "VllmConfig",
    kv_cache_groups: list["KVCacheGroupSpec"],
    num_blocks: int,
) -> "KVCacheConfig":
    """Materialize the groups for exactly ``num_blocks`` blocks.

    Each device backing's size is ``bytes_per_block * num_blocks``. All
    device tensors share one backing allocation unless the builder
    overrides ``_get_pool_bytes_per_block``. Host-resident tensors, such as
    HiSparse source storage, are excluded from the device byte accounting
    and sized independently.
    """
    raise NotImplementedError

get_kv_cache_groups(vllm_config, kv_cache_spec) abstractmethod

Split the given layers into KV cache groups.

Planning passes all workers' layers; profiling passes one worker's. Core rejects a result that does not put each layer in exactly one group.

The default builder honors a model's :class:~vllm.model_executor.models.interfaces.SupportsKVCacheGrouping declaration here before falling back to the default grouping; a platform-specific builder decides its own grouping.

Source code in vllm/v1/core/kv_cache_config_builder.py 
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
@abstractmethod
def get_kv_cache_groups(
    self,
    vllm_config: "VllmConfig",
    kv_cache_spec: dict[str, "KVCacheSpec"],
) -> list["KVCacheGroupSpec"]:
    """Split the given layers into KV cache groups.

    Planning passes all workers' layers; profiling passes one worker's.
    Core rejects a result that does not put each layer in exactly one group.

    The default builder honors a model's
    :class:`~vllm.model_executor.models.interfaces.SupportsKVCacheGrouping`
    declaration here before falling back to the default grouping; a
    platform-specific builder decides its own grouping.
    """
    raise NotImplementedError

get_kv_cache_config_builder(vllm_config)

Resolve the builder selected by the current platform.

Resolution priority is owned by the platform hook (:meth:vllm.platforms.interface.Platform.get_kv_cache_config_builder_cls). Builders are stateless (every method takes vllm_config), so a fresh instance is returned per call and engines with different configs in the same process always get the correct builder.

Source code in vllm/v1/core/kv_cache_config_builder.py 
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
def get_kv_cache_config_builder(
    vllm_config: "VllmConfig",
) -> KVCacheConfigBuilder:
    """Resolve the builder selected by the current platform.

    Resolution priority is owned by the platform hook
    (:meth:`vllm.platforms.interface.Platform.get_kv_cache_config_builder_cls`).
    Builders are stateless (every method takes ``vllm_config``), so a fresh
    instance is returned per call and engines with different configs in the
    same process always get the correct builder.
    """
    from vllm.platforms import current_platform

    builder_cls = resolve_obj_by_qualname(
        current_platform.get_kv_cache_config_builder_cls(vllm_config)
    )
    return builder_cls()