Upload folder using huggingface_hub

README.md

@@ -0,0 +1,42 @@
+---
+language: en
+license: mit
+library_name: pytorch
+tags:
+- deepseek
+- moe
+- mla
+- tiny-stories
+- text-generation
+model_name: DeepSeekV3
+base_model: deepseek-v3
+---
+
+# DeepSeekV3
+
+Implementation of the **DeepSeek-V3** architecture (8x2 MoE).
+
+## Architecture Details
+- **Architecture**: Mixture of Experts (MoE) + Multi-Head Latent Attention (MLA)
+- **Parameters**: ~196 million
+- **Layers**: 6
+- **Hidden Dimension**: 512
+- **Experts**: 8 routed experts (Top-2) + 1 Shared Expert
+- **Attention**: MLA (Multi-Head Latent Attention) with 8 heads and 64-dim Latent Space
+- **Position Embeddings**: Sinusoidal
+
+## Training
+- **Dataset**: TinyStories
+- **Platform**: Kaggle (2x T4 GPUs)
+- **Optimizer**: AdamW
+
+## How to use
+This model uses a custom architecture implementation. To load it, you can use the `state_dict` found in `model.safetensors` alongside a loader.
+
+```python
+from safetensors.torch import load_file
+weights = load_file("model.safetensors")
+```
+
+## Dataset
+Trained on the TinyStories dataset.

config.json

@@ -0,0 +1,17 @@
+{
+  "architectures": [
+    "DeepseekV3ForCausalLM"
+  ],
+  "model_type": "deepseek_v3_mini",
+  "hidden_size": 512,
+  "intermediate_size": 1365,
+  "num_attention_heads": 8,
+  "num_hidden_layers": 6,
+  "n_routed_experts": 8,
+  "num_experts_per_tok": 2,
+  "latent_disturation_dim": 64,
+  "vocab_size": 50257,
+  "max_position_embeddings": 256,
+  "rms_norm_eps": 1e-06,
+  "torch_dtype": "float32"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3d74575a37e1b268088cb4925defe3405c43886773b2f8731c421c0615d3b69d
+size 679343512