IvmeLabs/Ivme-Coder-v1

İvme-Coder-v1 (codename: Otter 1)

İvme (Turkish: acceleration) is a series of stupidly small language models built to punch above their weight. Ivme-Coder-v1 is the first release in a new branch of the family: a ~50M parameter decoder-only model trained from scratch on Python source code.

---

Model Details

Parameter	Value
Architecture	Decoder-only transformer, dense (no loops, no exotic recurrence)
Parameters	~50M
Layers	12
Hidden dim	512
FFN	SwiGLU (~2.67× expansion)
Attention heads	8, full attention (no GQA)
Context length	1024 tokens
Vocab size	16,000 (custom byte-level BPE, trained on Python source)
Positional encoding	RoPE (θ=10,000)
Normalization	RMSNorm (pre-norm)
Embeddings	Tied input/output
Biases	None

Coder's architecture deliberately mirrors the same core recipe as the Conversate line — RoPE, SwiGLU, RMSNorm, tied embeddings, Muon-optimized body weights. The point of keeping the recipe constant across both lines is to isolate what changes with domain-specific data and a narrower target distribution, not to introduce a second set of untested architectural choices at the same time.

---

Training

Data

Coder was trained on a filtered slice of bigcode/starcoderdata (Python configuration only), the same decontaminated, permissively-licensed corpus underlying the original StarCoder models. Raw files were passed through length and quality heuristics — dropping empty or oversized files, minified or generated-looking source, and files with abnormal symbol density — before being packed into fixed-length training shards.

Setting	Value
Source	bigcode/starcoderdata, Python config
Filtering	Length caps, minified/generated-file heuristics, symbol-density threshold
Total tokens	~2.5B

Unlike a general-text model, code has a far narrower and more rigid target distribution — Python's syntax is small, unambiguous, and repetitive relative to open-domain English. That changes the token math: Coder deliberately targets well past strict compute-optimal scaling for a model this size, but nowhere near the scale a general-purpose model in the same parameter class would need, since there is comparatively little left to learn once syntax, common idioms, and standard-library usage patterns are well covered.

Hyperparameters

Setting	Value
Optimizer	Muon (body weights) + AdamW (embeddings, norms)
LR schedule	Warmup–Stable–Decay (WSD)
Weight decay	0.1
Gradient clipping	1.0
Batch size	16 sequences × 12 grad-accum steps × 1024 tokens (196,608 tokens/step)
Total steps	12,715
Precision	bfloat16 (autocast), fp32 stored weights
Compilation	torch.compile
Final weights	EMA (β=0.999) of the training trajectory

Hardware

Trained on an RTX PRO 6000 Blackwell GPU on Google Colab, sustaining a stable ~517,000 tokens/second throughout training — the full run completed in under two hours.

---

Does it actually write code?

Short answer: it writes code-shaped text, reliably. It does not reliably write correct code. At 50M parameters and ~2.5B training tokens, Coder has clearly internalized Python's surface structure — indentation, control flow, function and class definitions, docstring conventions, common standard-library idioms — but it has not learned to reason about what a function is supposed to compute. Expect syntactically valid, plausible-looking completions that are frequently wrong on inspection: incorrect recursive logic, docstrings that don't match the implementation below them, redefinitions of the same method twice in one class, and functions that trail off into a related-but-different function rather than terminating cleanly.

This is the expected and honest ceiling for a model this size trained this way — it is closer to a highly fluent autocomplete for Python's shape than to a coding assistant, and it should be treated accordingly.

Sample output, EMA weights, temperature 0.8, top_k 50:

Prompt: def fibonacci(n):

def fibonacci(n): # Fibonacci for 5.2 if n < 3: return n

    if n % 2 == 0:
        return n

    return n // 2

from collections import OrderedDict # For compatibility with Python 2.6

def reverse(self, n):
    # Fibonacci for 5.2
    if n > 3:
        return n

    if n % 2 == 0:
        return n

    if n % 3 == 0:
        return n

    return n // 4

The indentation, control-flow shape, and import placement are all locally plausible. The logic is not — this is not a working Fibonacci implementation, and the model has no way to know that. This gap between fluency and correctness is the central thing to understand about Coder before using it for anything beyond exploration.

---

Limitations

• Base model only, not instruction tuned — completes code, does not follow instructions
• Learns Python syntax and idiom well; does not reliably produce logically correct or semantically meaningful code
• 1024 token context window
• Python only — no other languages in the training mix
• No retrieval-augmented or fine-tuned variant currently exists; this is a raw pretraining checkpoint

---

Inference

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "IvmeLabs/Ivme-Coder-v1", trust_remote_code=True, dtype=torch.float32,
)
tokenizer = AutoTokenizer.from_pretrained("IvmeLabs/Ivme-Coder-v1", trust_remote_code=True)
model.eval()

inputs = tokenizer("def fibonacci(n):", return_tensors="pt")
out = model.generate(
    **inputs, max_new_tokens=200, do_sample=True,
    temperature=0.8, top_k=50, pad_token_id=tokenizer.pad_token_id,
)
print(tokenizer.decode(out[0], skip_special_tokens=True))

trust_remote_code=True is required (custom architecture: RoPE + SwiGLU + RMSNorm dense decoder).

Note on batch generation: use left-padding (tokenizer.padding_side = "left") — the model does not use an explicit attention mask over padded positions, so right-padding within a batch will give incorrect results.

---

What's Next

Coder v1 establishes the animal line and the basic recipe: same architecture family as the fruit line, redirected at a single narrow domain. Future directions under consideration include a larger token budget to test where the syntax-fluency ceiling actually sits for a model this size, an instruction-tuned variant for actual completion/chat use rather than raw continuation, and eventually a second animal-line entry in a different narrow domain to test how well the recipe generalizes across disciplines, not just across scale.

You can check our other upcoming models on our organization card!

---

Citation

@misc{ivme-coder-v1,
  author       = {IvmeLabs},
  title        = {İvme-Coder-v1 (Otter 1)},
  year         = {2026},
  publisher    = {Hugging Face},
  url          = {https://huggingface.co/IvmeLabs/Ivme-Coder-v1}
}

---

Built by IvmeLabs. Small models, deliberate choices.