Skip to content

vllm.utils.humming

Lazy facade for the optional humming package.

vLLM code should import humming symbols from here so that import humming (which has import-time side effects) is deferred until first use. Add new symbols to both the TYPE_CHECKING imports and _EXPORTS, using "module.path:attr" or "module.path" for a whole-module re-export.

_patch_humming()

Make humming's per-call DeviceInfo(device) lookups traceable.

The forward path (LayerConfig.check_device, PDL heuristics) builds a DeviceInfo on every call, which breaks vLLM's fullgraph compile: Dynamo recurses forever in DeviceInfo.__getattr__ (vllm-project/humming#147) and, with that fixed (#148), still cannot trace the extension's _DeviceInfo constructor. Hand out one cached instance per device instead; prepare_layer_config creates it eagerly during weight processing.

TODO: remove after bumping humming to a release with vllm-project/humming#149.

Source code in vllm/utils/humming.py 
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
def _patch_humming() -> None:
    """Make humming's per-call ``DeviceInfo(device)`` lookups traceable.

    The forward path (``LayerConfig.check_device``, PDL heuristics) builds a
    ``DeviceInfo`` on every call, which breaks vLLM's fullgraph compile:
    Dynamo recurses forever in ``DeviceInfo.__getattr__``
    (vllm-project/humming#147) and, with that fixed (#148), still cannot
    trace the extension's ``_DeviceInfo`` constructor. Hand out one cached
    instance per device instead; ``prepare_layer_config`` creates it eagerly
    during weight processing.

    TODO: remove after bumping humming to a release with
    vllm-project/humming#149.
    """
    global _patched
    if _patched:
        return
    _patched = True

    import humming.config.config
    import humming.transform
    import humming.tune
    import humming.tune.process_input
    import humming.utils.smem
    from humming.device import DeviceInfo, get_device_index

    def __getattr__(self, name: str) -> Any:
        if name.startswith("_"):
            raise AttributeError(
                f"{type(self).__name__!r} object has no attribute {name!r}"
            )
        return getattr(self._get_value(), name)

    DeviceInfo.__getattr__ = __getattr__

    device_infos: dict[int, DeviceInfo] = {}

    def cached_device_info(index: int | torch.device | None = None) -> DeviceInfo:
        if isinstance(index, torch.device) and index.type != "cuda":
            return DeviceInfo(index)
        device_index = get_device_index(index)
        if device_index not in device_infos:
            device_infos[device_index] = DeviceInfo(device_index)
        return device_infos[device_index]

    for module in (
        humming.config.config,
        humming.transform,
        humming.tune,
        humming.tune.process_input,
        humming.utils.smem,
    ):
        if getattr(module, "DeviceInfo", None) is DeviceInfo:
            module.DeviceInfo = cached_device_info