failed09/bashkir-roberta

BashkirRoBERTa

A masked language model for Bashkir, for fill-mask, spellchecking and foundation fine-tuning.

Overview

A masked language model for Bashkir. Given a sentence with one [MASK] token, it predicts the most probable missing Bashkir token from context. The model is useful for fill-mask experiments, spellchecking and as a foundation for further fine-tuning. It preserves a custom Pre-LayerNorm architecture rather than the stock post-LayerNorm RoBERTa implementation.

At a glance
Task	Masked language modelling / fill-mask
Default artifact	model.safetensors (Transformers) or onnx/model_int8.onnx (ONNX)
Source	A monolingual Bashkir-language dataset
Version / license	v1 / custom terms (other)

Contents

Files and Configurations

File	Purpose	Size
model.safetensors	PyTorch weights for Transformers	200.2 MB
onnx/model_fp16.onnx	FP16 ONNX model for GPU / DirectML	121.2 MB
onnx/model_int8.onnx	INT8 ONNX model for fast CPU / mobile	60.9 MB
spm_bashkir_bert_16k.model	SentencePiece tokenizer	—
config.json	Model configuration (auto_map for custom code)	—
configuration_bashkir_roberta.py, modeling_bashkir_roberta.py, tokenization_bashkir_roberta.py	Custom Pre-LayerNorm implementation	—
tokenizer_config.json	Tokenizer configuration	—
META.json	Release passport and artifact hashes	—
SHA256SUMS	Release checksums	—

Model Architecture

Property	Value
Task	Masked language modelling / fill-mask
Architecture	Pre-LayerNorm Transformer encoder
Transformer blocks	8
Hidden size / attention heads	640 / 10
Feed-forward size	2,560
Context window	256 subword tokens
Parameters	50.04M
Tokenizer	SentencePiece BPE, 16,384 tokens

The output embedding matrix is tied to the input word embeddings. Token IDs are fixed: <pad> 0, <unk> 1, <s> 2, </s> 3, [CLS] 4, [SEP] 5 and [MASK] 6.

Examples

Outputs from the INT8 ONNX model on CPU:

Input	Top prediction
Мин башҡорт телен [MASK].	яратам
Башҡортостан — беҙҙең [MASK].	республика
Өфө — ҙур [MASK].	ҡала
Бөгөн Өфөлә яңы [MASK] асылды.	мәсет

Method

The model was pretrained with dynamic masked-language modelling on a monolingual Bashkir-language dataset assembled from encyclopedic, periodical and literary sources. The source texts are not distributed in this repository.

Evaluation

On a held-out Bashkir encyclopedic evaluation set the project reports 24.7% top-1 and 54.0% top-5 accuracy for masked subword prediction. These are diagnostic MLM results, not a general-purpose language-understanding score: a mask may represent a whole word or a SentencePiece subword fragment.

Quality and Use

This is a research model, not a production language service. Fill-mask predictions are ranking suggestions that require context-appropriate review, especially for ambiguous or short contexts. The checkpoint is released under custom terms while the source-rights audit is completed.

Limitations

• Diagnostic MLM accuracy only; not fine-tuned for any downstream task.
• A mask may correspond to a partial subword, not always a full word.
• Predictions reflect the training corpus and may prefer frequent or encyclopedic phrasing.
• No training texts are redistributed; provenance or removal requests go through the maintainer.

Usage

pip install transformers torch huggingface_hub

PyTorch (Transformers), which requires trust_remote_code=True because of the custom Pre-LayerNorm architecture:

from transformers import AutoModelForMaskedLM, AutoTokenizer

repo_id = "failed09/bashkir-roberta"
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForMaskedLM.from_pretrained(repo_id, trust_remote_code=True)

inputs = tokenizer("Мин башҡорт телен [MASK].", return_tensors="pt")
logits = model(**inputs).logits
mask_index = inputs["input_ids"][0].tolist().index(tokenizer.mask_token_id)
prediction_id = logits[0, mask_index].argmax().item()
print(tokenizer.decode([prediction_id]))  # яратам

ONNX Runtime for CPU and edge deployment:

import numpy as np
import onnxruntime as ort
import sentencepiece as spm
from huggingface_hub import hf_hub_download

model_path = hf_hub_download("failed09/bashkir-roberta", "onnx/model_int8.onnx")
sp_path = hf_hub_download("failed09/bashkir-roberta", "spm_bashkir_bert_16k.model")

session = ort.InferenceSession(model_path, providers=["CPUExecutionProvider"])
sp = spm.SentencePieceProcessor(model_file=sp_path)

tokens = [2] + sp.encode("Мин башҡорт телен ") + [6] + sp.encode(".") + [3]
mask_idx = tokens.index(6)
logits = session.run(None, {"input_ids": np.array([tokens], dtype=np.int64)})[0][0, mask_idx]
top_tokens = np.argsort(logits)[::-1][:5]
print([sp.decode([int(t)]) for t in top_tokens])  # ['яратам', 'беләм', 'өйрәнә', ...]

License

The checkpoint is released under custom terms (other on the Hub) while the source-rights audit is completed. No training texts are redistributed. For provenance or removal requests, contact the maintainer through the Hub.

Citation

@software{failed09_bashkir_roberta_2026,
  title = {BashkirRoBERTa},
  author = {failed09},
  year = {2026},
  publisher = {Hugging Face},
  url = {https://huggingface.co/failed09/bashkir-roberta},
  note = {Masked language model for Bashkir}
}

Open Bashkir Data and Sources 🐝

This release is part of an open-source effort to support the development, preservation and practical use of the Bashkir language. Other related models, datasets and tools are available on the author's Hugging Face profile.

The author does not claim ownership or authorship of the source texts or other materials used to derive this release; rights and licensing remain with the original authors, publishers and dataset providers. Source texts are not redistributed in this repository, so users should follow the licenses and attribution requirements of the relevant upstream resources.