Instructions to use zai-org/GLM-5.2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use zai-org/GLM-5.2 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="zai-org/GLM-5.2")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("zai-org/GLM-5.2")
model = AutoModelForCausalLM.from_pretrained("zai-org/GLM-5.2")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
HuggingChat
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use zai-org/GLM-5.2 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "zai-org/GLM-5.2"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "zai-org/GLM-5.2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/zai-org/GLM-5.2

SGLang

How to use zai-org/GLM-5.2 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "zai-org/GLM-5.2" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "zai-org/GLM-5.2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "zai-org/GLM-5.2" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "zai-org/GLM-5.2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use zai-org/GLM-5.2 with Docker Model Runner:
```
docker model run hf.co/zai-org/GLM-5.2
```

Anyone tried this to run only on standard PC ram only?

#46

by gemstonebro - opened 3 days ago

Discussion

gemstonebro

3 days ago

Anyone tried this to run only on standard PC ram only? Like basic CPU workstation mobo and ram.

If yes how much tokens/s can this go during long context? Please please tell me!!!

OblxDev

3 days ago

Basic cpu and basic ram will freeze/OOM/fry if you try on a consumer grade machine.
The super bare minimal setup I theorically found is 2x3090 24Gb Vram + 256Gb ram to split the KV and be able to run at... 1-2tps (in the minimal compressed version).

Didn't test it yet because I only have 128Gb max on my biggest machine.

Your best option to have it unlimited at home is either to have a big cluster of gpus or a friend with big gpus, or use cloud providers. Enjoy !

gemstonebro

3 days ago

Basic cpu and basic ram will freeze/OOM/fry if you try on a consumer grade machine.
The super bare minimal setup I theorically found is 2x3090 24Gb Vram + 256Gb ram to split the KV and be able to run at... 1-2tps (in the minimal compressed version).

Didn't test it yet because I only have 128Gb max on my biggest machine.

Your best option to have it unlimited at home is either to have a big cluster of gpus or a friend with big gpus, or use cloud providers. Enjoy !

Hmm thanks! Okay then i suppose i will drop "consumer grade" hopes.. How do i know what item is "not consumer grade" based on price? must have "workstation or server word in it?"

OblxDev

3 days ago

Well, a 3090 is technically consumer grade, A 3060 12Gb too (avoid 4000 serie unless super incredible price), then there is the 5090.

Regarding prices, there exists three eras ^^
Pre crypto prices
Pre AI prices
Current prices, inheriting from the previous ones. Don't mistake me, rise in prices means the demand is super high (combined to scarcity of silicon-based components) and that people are actually willing to fiddle with AI at home. Check your local second hand market for 3090, they are sometimes cheaper wrapped in a fully working system 😎

Then there are some homelabs with 8x3090 which is still consumer grade !

To sumup the difference between consumer/cluster is indeed the price (juicy 8xMi300X or HB100 or HB200) but the physical hardware that is ahead of consumer one. Bandwith, memory correction, network, monitoring, cooling, power consumption... They are all on an other level.

(If you have brave browser, Ctrl + B then paste the technical stuff to the chat, or copy paste to your favorite chatbot and request more insight regarding consumer/cluster/datacenter based on my answer).

Enjoy the big numbers 🤓

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment