In March 2024, xAI released the model weights and architecture of Grok-1, the base model behind its Grok chat assistant, under the Apache 2.0 license. At 314 billion parameters it was, at release, among the largest models with openly available weights, published via the company’s xai-org/grok-1 GitHub repository with a JAX implementation for loading and inference.
Grok-1 is a mixture-of-experts model: 8 experts with 2 active per token, 64 layers, and an embedding dimension of 6,144, using a 131,072-token SentencePiece vocabulary. The released checkpoint was the raw pre-trained base model, not fine-tuned for any particular task such as dialogue, and the repository’s README candidly noted that its reference MoE implementation was not optimized and that the model’s scale demanded substantial hardware.
The release fit a broader 2024 pattern of frontier labs putting large model weights into the open, alongside Meta’s Llama series and Mistral’s Mixtral. For xAI, a company barely a year old, open-sourcing Grok-1 was also a competitive and political statement about openness in AI at a moment when most of the largest models remained closed.