Partial RoPE Analysis

aflah 's Collections

Geoanthropology Q&A Models

updated 7 days ago

Models trained via GPT-NeoX for studying Partial RoPE.

Upvote

Fractional Rotation, Full Potential? Investigating Performance and Convergence of Partial RoPE

Paper • 2603.11611 • Published Mar 12
aflah/Llama1BxFW_Edux2048x0pct

Updated 13 days ago

Note - Model Based on: Llama 1B - Arch Type: Sequential - Dataset: FineWeb-Edu (FW_Edu) - Sequence Length: 2048 - Partial Rope Factor: 0%
aflah/Llama1BxFW_Edux2048x4pct

Updated 13 days ago

Note - Model Based on: Llama 1B - Arch Type: Sequential - Dataset: FineWeb-Edu (FW_Edu) - Sequence Length: 2048 - Partial Rope Factor: 4%
aflah/Llama1BxFW_Edux2048x10pct

Updated 13 days ago

Note - Model Based on: Llama 1B - Arch Type: Sequential - Dataset: FineWeb-Edu (FW_Edu) - Sequence Length: 2048 - Partial Rope Factor: 10%
aflah/Llama1BxFW_Edux2048x25pct

Updated 13 days ago

Note - Model Based on: Llama 1B - Arch Type: Sequential - Dataset: FineWeb-Edu (FW_Edu) - Sequence Length: 2048 - Partial Rope Factor: 25%
aflah/Llama1BxFW_Edux2048x50pct

Updated 13 days ago

Note - Model Based on: Llama 1B - Arch Type: Sequential - Dataset: FineWeb-Edu (FW_Edu) - Sequence Length: 2048 - Partial Rope Factor: 50%
aflah/Llama1BxFW_Edux2048x75pct

Updated 13 days ago

Note - Model Based on: Llama 1B - Arch Type: Sequential - Dataset: FineWeb-Edu (FW_Edu) - Sequence Length: 2048 - Partial Rope Factor: 75%
aflah/Llama1BxFW_Edux2048x100pct

Updated 13 days ago

Note - Model Based on: Llama 1B - Arch Type: Sequential - Dataset: FineWeb-Edu (FW_Edu) - Sequence Length: 2048 - Partial Rope Factor: 100%
aflah/Llama1BxFWx2048x0pct

Updated 13 days ago

Note - Model Based on: Llama 1B - Arch Type: Sequential - Dataset: FineWeb (FW) - Sequence Length: 2048 - Partial Rope Factor: 0%
aflah/Llama1BxFWx2048x4pct

Updated 13 days ago

Note - Model Based on: Llama 1B - Arch Type: Sequential - Dataset: FineWeb (FW) - Sequence Length: 2048 - Partial Rope Factor: 4%
aflah/Llama1BxFWx2048x10pct

Updated 13 days ago

Note - Model Based on: Llama 1B - Arch Type: Sequential - Dataset: FineWeb (FW) - Sequence Length: 2048 - Partial Rope Factor: 10%
aflah/Llama1BxFWx2048x25pct

Updated 13 days ago

Note - Model Based on: Llama 1B - Arch Type: Sequential - Dataset: FineWeb (FW) - Sequence Length: 2048 - Partial Rope Factor: 25%
aflah/Llama1BxFWx2048x50pct

Updated 13 days ago

Note - Model Based on: Llama 1B - Arch Type: Sequential - Dataset: FineWeb (FW) - Sequence Length: 2048 - Partial Rope Factor: 50%
aflah/Llama1BxFWx2048x75pct

Updated 13 days ago

Note - Model Based on: Llama 1B - Arch Type: Sequential - Dataset: FineWeb (FW) - Sequence Length: 2048 - Partial Rope Factor: 75%
aflah/Llama1BxFWx2048x100pct

Updated 13 days ago

Note - Model Based on: Llama 1B - Arch Type: Sequential - Dataset: FineWeb (FW) - Sequence Length: 2048 - Partial Rope Factor: 100%
aflah/Llama1BxFWx1024x0pct

Updated 13 days ago

Note - Model Based on: Llama 1B - Arch Type: Sequential - Dataset: FineWeb (FW) - Sequence Length: 1024 - Partial Rope Factor: 0%
aflah/Llama1BxFWx1024x4pct

Updated 13 days ago

Note - Model Based on: Llama 1B - Arch Type: Sequential - Dataset: FineWeb (FW) - Sequence Length: 1024 - Partial Rope Factor: 4%
aflah/Llama1BxFWx1024x10pct

Updated 13 days ago

Note - Model Based on: Llama 1B - Arch Type: Sequential - Dataset: FineWeb (FW) - Sequence Length: 1024 - Partial Rope Factor: 10%
aflah/Llama1BxFWx1024x25pct

Updated 13 days ago

Note - Model Based on: Llama 1B - Arch Type: Sequential - Dataset: FineWeb (FW) - Sequence Length: 1024 - Partial Rope Factor: 25%
aflah/Llama1BxFWx1024x50pct

Updated 13 days ago

Note - Model Based on: Llama 1B - Arch Type: Sequential - Dataset: FineWeb (FW) - Sequence Length: 1024 - Partial Rope Factor: 50%
aflah/Llama1BxFWx1024x75pct

Updated 13 days ago

Note - Model Based on: Llama 1B - Arch Type: Sequential - Dataset: FineWeb (FW) - Sequence Length: 1024 - Partial Rope Factor: 75%
aflah/Llama1BxFWx1024x100pct

Updated 13 days ago

Note - Model Based on: Llama 1B - Arch Type: Sequential - Dataset: FineWeb (FW) - Sequence Length: 1024 - Partial Rope Factor: 100%
aflah/Llama1BxFWx4096x0pct

Updated 13 days ago

Note - Model Based on: Llama 1B - Arch Type: Sequential - Dataset: FineWeb (FW) - Sequence Length: 4096 - Partial Rope Factor: 0%
aflah/Llama1BxFWx4096x4pct

Updated 13 days ago

Note - Model Based on: Llama 1B - Arch Type: Sequential - Dataset: FineWeb (FW) - Sequence Length: 4096 - Partial Rope Factor: 4%
aflah/Llama1BxFWx4096x10pct

Updated 13 days ago

Note - Model Based on: Llama 1B - Arch Type: Sequential - Dataset: FineWeb (FW) - Sequence Length: 4096 - Partial Rope Factor: 10%
aflah/Llama1BxFWx4096x25pct

Updated 13 days ago

Note - Model Based on: Llama 1B - Arch Type: Sequential - Dataset: FineWeb (FW) - Sequence Length: 4096 - Partial Rope Factor: 25%
aflah/Llama1BxFWx4096x50pct

Updated 13 days ago

Note - Model Based on: Llama 1B - Arch Type: Sequential - Dataset: FineWeb (FW) - Sequence Length: 4096 - Partial Rope Factor: 50%
aflah/Llama1BxFWx4096x75pct

Updated 13 days ago

Note - Model Based on: Llama 1B - Arch Type: Sequential - Dataset: FineWeb (FW) - Sequence Length: 4096 - Partial Rope Factor: 75%
aflah/Llama1BxFWx4096x100pct

Updated 13 days ago

Note - Model Based on: Llama 1B - Arch Type: Sequential - Dataset: FineWeb (FW) - Sequence Length: 4096 - Partial Rope Factor: 100%
aflah/Llama1BxFWx8192x0pct

Updated 13 days ago

Note - Model Based on: Llama 1B - Arch Type: Sequential - Dataset: FineWeb (FW) - Sequence Length: 8192 - Partial Rope Factor: 0%
aflah/Llama1BxFWx8192x0pctxqk_norm

Updated 13 days ago

Note - Model Based on: Llama 1B - Arch Type: Sequential - Dataset: FineWeb (FW) - Sequence Length: 8192 - Partial Rope Factor: 0% - Metadata: QK Norm
aflah/Llama1BxFWx8192x4pct

Updated 13 days ago

Note - Model Based on: Llama 1B - Arch Type: Sequential - Dataset: FineWeb (FW) - Sequence Length: 8192 - Partial Rope Factor: 4%
aflah/Llama1BxFWx8192x10pct

Updated 13 days ago

Note - Model Based on: Llama 1B - Arch Type: Sequential - Dataset: FineWeb (FW) - Sequence Length: 8192 - Partial Rope Factor: 10%
aflah/Llama1BxFWx8192x25pct

Updated 13 days ago

Note - Model Based on: Llama 1B - Arch Type: Sequential - Dataset: FineWeb (FW) - Sequence Length: 8192 - Partial Rope Factor: 25%
aflah/Llama1BxFWx8192x50pct

Updated 13 days ago

Note - Model Based on: Llama 1B - Arch Type: Sequential - Dataset: FineWeb (FW) - Sequence Length: 8192 - Partial Rope Factor: 50%
aflah/Llama1BxFWx8192x75pct

Updated 13 days ago

Note - Model Based on: Llama 1B - Arch Type: Sequential - Dataset: FineWeb (FW) - Sequence Length: 8192 - Partial Rope Factor: 75%
aflah/Llama1BxFWx8192x100pct

Updated 13 days ago

Note - Model Based on: Llama 1B - Arch Type: Sequential - Dataset: FineWeb (FW) - Sequence Length: 8192 - Partial Rope Factor: 100%
aflah/Llama8BxFWx2048x0pct

Updated 13 days ago

Note - Model Based on: Llama 8B - Arch Type: Sequential - Dataset: FineWeb (FW) - Sequence Length: 2048 - Partial Rope Factor: 0%
aflah/Llama8BxFWx2048x10pct

Updated 13 days ago

Note - Model Based on: Llama 8B - Arch Type: Sequential - Dataset: FineWeb (FW) - Sequence Length: 2048 - Partial Rope Factor: 10%
aflah/Llama8BxFWx2048x25pct

Updated 13 days ago

Note - Model Based on: Llama 8B - Arch Type: Sequential - Dataset: FineWeb (FW) - Sequence Length: 2048 - Partial Rope Factor: 25%
aflah/Llama8BxFWx2048x50pct

Updated 13 days ago

Note - Model Based on: Llama 8B - Arch Type: Sequential - Dataset: FineWeb (FW) - Sequence Length: 2048 - Partial Rope Factor: 50%
aflah/Llama8BxFWx2048x75pct

Updated 13 days ago

Note - Model Based on: Llama 8B - Arch Type: Sequential - Dataset: FineWeb (FW) - Sequence Length: 2048 - Partial Rope Factor: 75%
aflah/Llama8BxFWx2048x100pct

Updated 13 days ago

Note - Model Based on: Llama 8B - Arch Type: Sequential - Dataset: FineWeb (FW) - Sequence Length: 2048 - Partial Rope Factor: 100%
aflah/Pythia1BxFW_Edux2048x0pct

Updated 13 days ago

Note - Model Based on: Pythia 1B - Arch Type: Parallel - Dataset: FineWeb-Edu (FW_Edu) - Sequence Length: 2048 - Partial Rope Factor: 0%
aflah/Pythia1BxFW_Edux2048x0pctxqk_norm

Updated 13 days ago

Note - Model Based on: Pythia 1B - Arch Type: Parallel - Dataset: FineWeb-Edu (FW_Edu) - Sequence Length: 2048 - Partial Rope Factor: 0% - Metadata: QK Norm
aflah/Pythia1BxFW_Edux2048x0pctxlr_4E-5

Updated 13 days ago

Note - Model Based on: Pythia 1B - Arch Type: Parallel - Dataset: FineWeb-Edu (FW_Edu) - Sequence Length: 2048 - Partial Rope Factor: 0% - Metadata: Learning Rate: 4E-5
aflah/Pythia1BxFW_Edux2048x0pctxlr_4E-3

Updated 13 days ago

Note - Model Based on: Pythia 1B - Arch Type: Parallel - Dataset: FineWeb-Edu (FW_Edu) - Sequence Length: 2048 - Partial Rope Factor: 0% - Metadata: Learning Rate: 4E-3
aflah/Pythia1BxFW_Edux2048x0pctxseed_175

Updated 13 days ago

Note - Model Based on: Pythia 1B - Arch Type: Parallel - Dataset: FineWeb-Edu (FW_Edu) - Sequence Length: 2048 - Partial Rope Factor: 0% - Metadata: Seed: 175
aflah/Pythia1BxFW_Edux2048x0pctxseed_238

Updated 13 days ago

Note - Model Based on: Pythia 1B - Arch Type: Parallel - Dataset: FineWeb-Edu (FW_Edu) - Sequence Length: 2048 - Partial Rope Factor: 0% - Metadata: Seed: 238
aflah/Pythia1BxFW_Edux2048x0pctxseed_953

Updated 13 days ago

Note - Model Based on: Pythia 1B - Arch Type: Parallel - Dataset: FineWeb-Edu (FW_Edu) - Sequence Length: 2048 - Partial Rope Factor: 0% - Metadata: Seed: 953
aflah/Pythia1BxFW_Edux2048x1pct

Updated 13 days ago

Note - Model Based on: Pythia 1B - Arch Type: Parallel - Dataset: FineWeb-Edu (FW_Edu) - Sequence Length: 2048 - Partial Rope Factor: 1%
aflah/Pythia1BxFW_Edux2048x10pct

Updated 13 days ago

Note - Model Based on: Pythia 1B - Arch Type: Parallel - Dataset: FineWeb-Edu (FW_Edu) - Sequence Length: 2048 - Partial Rope Factor: 10%
aflah/Pythia1BxFW_Edux2048x25pct

Updated 13 days ago

Note - Model Based on: Pythia 1B - Arch Type: Parallel - Dataset: FineWeb-Edu (FW_Edu) - Sequence Length: 2048 - Partial Rope Factor: 25%
aflah/Pythia1BxFW_Edux2048x50pct

Updated 13 days ago

Note - Model Based on: Pythia 1B - Arch Type: Parallel - Dataset: FineWeb-Edu (FW_Edu) - Sequence Length: 2048 - Partial Rope Factor: 50%
aflah/Pythia1BxFW_Edux2048x75pct

Updated 13 days ago

Note - Model Based on: Pythia 1B - Arch Type: Parallel - Dataset: FineWeb-Edu (FW_Edu) - Sequence Length: 2048 - Partial Rope Factor: 75%
aflah/Pythia1BxFW_Edux2048x100pct

Updated 13 days ago

Note - Model Based on: Pythia 1B - Arch Type: Parallel - Dataset: FineWeb-Edu (FW_Edu) - Sequence Length: 2048 - Partial Rope Factor: 100%

Upvote