tencent

/

Hy-MT2-30B-A3B

like 475

Follow

Tencent 11.3k

It would be cool if the unsloth team released a GGUF version.

llama.cpp must be supported first. Then gguf

q8 pls

Team mradermacher agreed to quantize it, but it failed the pipeline as of now: https://huggingface.co/mradermacher/model_requests/discussions/2423

Before you get excited, read this:
https://huggingface.co/tencent/Hy-MT2-7B-GGUF/discussions/3

great

I made a q4 for testing. feel free to check it out

Before getting supported by llama.cpp, you can try chatllm.cpp:

main_nim -m :hy-mt2:a3b -ngl all --max-length 10000 --multi -i
    ________          __  __    __    __  ___ 
   / ____/ /_  ____ _/ /_/ /   / /   /  |/  /_________  ____  
  / /   / __ \/ __ `/ __/ /   / /   / /|_/ // ___/ __ \/ __ \ 
 / /___/ / / / /_/ / /_/ /___/ /___/ /  / // /__/ /_/ / /_/ / 
 \____/_/ /_/\__,_/\__/_____/_____/_/  /_(_)___/ .___/ .___/  
You are served by HY-MT2,                     /_/   /_/       
with 30064725888 (3.5B active) parameters.

You  > 将以下文本翻译为English，注意只需要输出翻译后的结果，不要额外解释：

钟书到清华工作一年后，调任毛选翻译委员会的工作，住在城里，周末回校，仍兼管研究生。毛选翻译委员会的领导是徐永焕同志，介绍钟书做这份工作的是清华同学乔冠华同志。事定之日，晚饭后，有一位旧友特雇黄 包车从城里赶来祝贺。客去后，钟书惶恐地对我说：“他以为我要做‘南书房行走’了。这件事不是好做的，不求有功，但求无过。”
\.
A.I. > <think></think>One year after starting work at Tsinghua, Zhongshu was transferred to work for the Translation Committee for Mao Zedong’s Selected Works. He lived in the city and returned to campus on weekends; he still remained in charge of the graduate students. The head of the Translation Committee was Comrade Xu Yonghuan. It was Comrade Qiao Guanhua, a fellow Tsinghua alumnus, who recommended Zhongshu for this job. On the day everything was finalized, after dinner, an old friend specially hired a rickshaw to come from the city to offer his congratulations. After his friend left, Zhongshu said to me anxiously, “He thinks I’m going to become a ‘scribe in the Southern Study.’ This is not an easy task; one need not strive to achieve great accomplishments, but rather simply avoid making any mistakes.”

@tencent , There seems to be an update to llama cpp for creating gguf model 30B.
Check, please:
https://github.com/ggml-org/llama.cpp/pull/25395
Hy3:

This PR adds the MTP spec primarily for Hy3. For Hy-MT2-30B-A3B, MTP is likely unnecessary; do not include --spec-type draft-mtp if the GGUF does not contain the corresponding layer.