bottlecapai/ThinkingCap-Qwen3.6-27B | INT4 (W4A16)
#39
by INC4AI - opened
Pipeline Failure Report
Model: bottlecapai/ThinkingCap-Qwen3.6-27B
Quantization Scheme: INT4 (W4A16)
Failed Phase: quantize
Run ID: ThinkingCap-Qwen3.6-27B-AutoRound-W4A16-RTN
Error Category: out_of_memory
Full Error Log
22:05:50 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/bottlecapai/ThinkingCap-Qwen3.6-27B/2cbd89d3fff9274633aa7b979643c75a9a81cabd/tokenizer_config.json "HTTP/1.1 200 OK"
22:05:50 [INFO] HTTP Request: GET https://huggingface.co/api/resolve-cache/models/bottlecapai/ThinkingCap-Qwen3.6-27B/2cbd89d3fff9274633aa7b979643c75a9a81cabd/tokenizer_config.json "HTTP/1.1 200 OK"
22:05:50 [INFO] HTTP Request: HEAD https://huggingface.co/bottlecapai/ThinkingCap-Qwen3.6-27B/resolve/main/tokenizer_config.json "HTTP/1.1 307 Temporary Redirect"
22:05:51 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/bottlecapai/ThinkingCap-Qwen3.6-27B/2cbd89d3fff9274633aa7b979643c75a9a81cabd/tokenizer_config.json "HTTP/1.1 200 OK"
22:05:51 [INFO] HTTP Request: GET https://huggingface.co/api/models/bottlecapai/ThinkingCap-Qwen3.6-27B/tree/main/additional_chat_templates?recursive=false&expand=false "HTTP/1.1 404 Not Found"
22:05:51 [INFO] HTTP Request: GET https://huggingface.co/api/models/bottlecapai/ThinkingCap-Qwen3.6-27B/tree/main?recursive=true&expand=false "HTTP/1.1 200 OK"
22:05:51 [INFO] HTTP Request: HEAD https://huggingface.co/bottlecapai/ThinkingCap-Qwen3.6-27B/resolve/main/vocab.json "HTTP/1.1 307 Temporary Redirect"
22:05:51 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/bottlecapai/ThinkingCap-Qwen3.6-27B/2cbd89d3fff9274633aa7b979643c75a9a81cabd/vocab.json "HTTP/1.1 200 OK"
22:05:51 [INFO] HTTP Request: GET https://huggingface.co/api/resolve-cache/models/bottlecapai/ThinkingCap-Qwen3.6-27B/2cbd89d3fff9274633aa7b979643c75a9a81cabd/vocab.json "HTTP/1.1 200 OK"
22:05:51 [INFO] HTTP Request: HEAD https://huggingface.co/bottlecapai/ThinkingCap-Qwen3.6-27B/resolve/main/merges.txt "HTTP/1.1 307 Temporary Redirect"
22:05:51 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/bottlecapai/ThinkingCap-Qwen3.6-27B/2cbd89d3fff9274633aa7b979643c75a9a81cabd/merges.txt "HTTP/1.1 200 OK"
22:05:51 [INFO] HTTP Request: GET https://huggingface.co/api/resolve-cache/models/bottlecapai/ThinkingCap-Qwen3.6-27B/2cbd89d3fff9274633aa7b979643c75a9a81cabd/merges.txt "HTTP/1.1 200 OK"
22:05:51 [INFO] HTTP Request: HEAD https://huggingface.co/bottlecapai/ThinkingCap-Qwen3.6-27B/resolve/main/tokenizer.json "HTTP/1.1 302 Found"
22:05:52 [INFO] HTTP Request: HEAD https://huggingface.co/bottlecapai/ThinkingCap-Qwen3.6-27B/resolve/main/added_tokens.json "HTTP/1.1 404 Not Found"
22:05:52 [INFO] HTTP Request: HEAD https://huggingface.co/bottlecapai/ThinkingCap-Qwen3.6-27B/resolve/main/special_tokens_map.json "HTTP/1.1 404 Not Found"
22:05:52 [INFO] HTTP Request: HEAD https://huggingface.co/bottlecapai/ThinkingCap-Qwen3.6-27B/resolve/main/chat_template.jinja "HTTP/1.1 307 Temporary Redirect"
22:05:53 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/bottlecapai/ThinkingCap-Qwen3.6-27B/2cbd89d3fff9274633aa7b979643c75a9a81cabd/chat_template.jinja "HTTP/1.1 200 OK"
22:05:53 [INFO] HTTP Request: GET https://huggingface.co/api/resolve-cache/models/bottlecapai/ThinkingCap-Qwen3.6-27B/2cbd89d3fff9274633aa7b979643c75a9a81cabd/chat_template.jinja "HTTP/1.1 200 OK"
22:05:54 [INFO] HTTP Request: GET https://huggingface.co/api/models/bottlecapai/ThinkingCap-Qwen3.6-27B "HTTP/1.1 200 OK"
22:05:54 [INFO] Loading model...
22:05:54 [INFO] HTTP Request: HEAD https://huggingface.co/bottlecapai/ThinkingCap-Qwen3.6-27B/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
22:05:54 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/bottlecapai/ThinkingCap-Qwen3.6-27B/2cbd89d3fff9274633aa7b979643c75a9a81cabd/config.json "HTTP/1.1 200 OK"
22:05:54 [INFO] HTTP Request: HEAD https://huggingface.co/bottlecapai/ThinkingCap-Qwen3.6-27B/resolve/main/model.safetensors "HTTP/1.1 404 Not Found"
22:05:54 [INFO] HTTP Request: HEAD https://huggingface.co/bottlecapai/ThinkingCap-Qwen3.6-27B/resolve/main/model.safetensors.index.json "HTTP/1.1 307 Temporary Redirect"
22:05:55 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/bottlecapai/ThinkingCap-Qwen3.6-27B/2cbd89d3fff9274633aa7b979643c75a9a81cabd/model.safetensors.index.json "HTTP/1.1 200 OK"
22:05:55 [INFO] HTTP Request: GET https://huggingface.co/api/resolve-cache/models/bottlecapai/ThinkingCap-Qwen3.6-27B/2cbd89d3fff9274633aa7b979643c75a9a81cabd/model.safetensors.index.json "HTTP/1.1 200 OK"
22:05:55 [INFO] HTTP Request: GET https://huggingface.co/api/models/bottlecapai/ThinkingCap-Qwen3.6-27B/revision/main "HTTP/1.1 200 OK"
22:05:55 [INFO] HTTP Request: GET https://huggingface.co/api/models/bottlecapai/ThinkingCap-Qwen3.6-27B/tree/2cbd89d3fff9274633aa7b979643c75a9a81cabd?recursive=true&expand=false "HTTP/1.1 200 OK"
[transformers] The fast path is not available because one of the required library is not installed. Falling back to torch implementation. To install follow https://github.com/fla-org/flash-linear-attention#installation and https://github.com/Dao-AILab/causal-conv1d
22:07:21 [ERROR] Quantization failed: CUDA out of memory. Tried to allocate 170.00 MiB. GPU 0 has a total capacity of 44.52 GiB of which 72.25 MiB is free. Process 512171 has 44.44 GiB memory in use. Of the allocated memory 43.90 GiB is allocated by PyTorch, and 132.19 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
Traceback (most recent call last):
File "/root/_work/1/s/auto_quant/phases/quantize.py", line 479, in <module>
quantize(args)
File "/root/_work/1/s/auto_quant/phases/quantize.py", line 293, in quantize
model = AutoModelForCausalLM.from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/auto_round/utils/common.py", line 140, in patched
return underlying_func(klass, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 402, in from_pretrained
return model_class.from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/transformers/modeling_utils.py", line 4368, in from_pretrained
loading_info, disk_offload_index = cls._load_pretrained_model(model, state_dict, checkpoint_files, load_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/transformers/modeling_utils.py", line 4502, in _load_pretrained_model
loading_info, disk_offload_index = convert_and_load_state_dict_in_model(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/transformers/core_model_loading.py", line 1660, in convert_and_load_state_dict_in_model
realized_value = mapping.convert(
^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/transformers/core_model_loading.py", line 958, in convert
collected_tensors = self.materialize_tensors()
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/transformers/core_model_loading.py", line 922, in materialize_tensors
tensors = [future.result() for future in tensors if future.result() is not None]
^^^^^^^^^^^^^^^
File "/root/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/concurrent/futures/_base.py", line 456, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/root/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/root/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/concurrent/futures/thread.py", line 59, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/transformers/core_model_loading.py", line 1199, in _job
return _materialize_copy(tensor, device, dtype)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/transformers/core_model_loading.py", line 1185, in _materialize_copy
tensor = tensor.to(device=device, dtype=dtype)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 170.00 MiB. GPU 0 has a total capacity of 44.52 GiB of which 72.25 MiB is free. Process 512171 has 44.44 GiB memory in use. Of the allocated memory 43.90 GiB is allocated by PyTorch, and 132.19 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
Auto-generated by error_analysis pipeline. cc @lvkaokao