Jarrel Seah
jarrelscy
AI & ML interests
None yet
Recent Activity
updated a model 1 day ago
jarrelscy/Inkling-512k-NVFP4-AQLM-hybrid published a model 3 days ago
jarrelscy/Inkling-512k-NVFP4-AQLM-hybrid updated a model 6 days ago
jarrelscy/GLM-5.2-NVFP4-AQLM-hybridOrganizations
None yet
I am running in vllm 0.4.1 with 4 x gpus 24gb (A10G 24gb) = 96gb and eager mode and I am still out of memory, how? it should fit (like 87gb vram)
1
#3 opened about 2 years ago
by 
orel12
KeyError: 'model.layers.45.block_sparse_moe.gate.g_idx'
5
#2 opened about 2 years ago
by 
tutu329
KeyError: 'model.layers.45.block_sparse_moe.gate.g_idx'
5
#2 opened about 2 years ago
by tutu329