Marin

"I am not afraid of storms, for I am learning how to sail my ship."
– Louisa May Alcott

Marin is a research program, software platform, and community for the research and development of foundation models.

Marin's concern is training large language models. This includes data curation, transformation, filtering, tokenization, pretraining, posttraining, and evaluation. Beyond the artifacts, software, and infrastructure, behind these models, Marin is committed to openly sharing all of the process knowledge required to build these models.

Marin's core value is open development. We document our processes, experiments, and decisions as they happen. Every step, from raw data to the final model, is recorded. Failed experiments are part of that record.

Marin has also been used for building audio-text models, DNA, and protein models. We encourage this work through the use of Marin as a library, in marin/experiments.

Current work

Frontier mixture-of-experts

Our current focus is pretraining, from scratch, and posttraining a large (5e24 model-FLOPs, 500 billion+ total parameters) mixture-of-experts model to succeed on tasks of importance to scientists and researchers.

Scaling suite

Delphi is Marin's open scaling suite scaling a LLM recipe from 3e18 to 1e23 FLOPs, inspired by Pythia. It has three parts: a scaling recipe that maps compute budgets to model configurations, a scaling suite trained from that recipe on the Google TPU Research Cloud, and a scaling law that uses the smaller Delphi models to predict the larger ones.

We released:

Checkpoints for every run, available on Hugging Face at marin-community/delphi
Training mixture pipelines that deterministically reproduce the mix from the public Nemotron-CC, StarCoderData, and ProofPile 2 in the Marin repo
Recipe code as a forkable CompletedAdamHParams class in the Marin repo
Development methodology as the add_scaling_heuristic agent skill in the Marin repo
Plot-ready data for the Delphi figures, with one config per figure and a wandb_url on every row, at marin-community/delphi-blog-data

Progress was tracked in GitHub issue #1337.

Other learnings

Some additional consolidated learnings can be found on the Open Athena blog. A selection, below:

Cluster Scheduling with Iris · scheduling jobs across heterogeneous clusters
Improving our LLM Pretraining Efficiency · squeezing more throughput from pretraining
Scaling Laws That Extrapolate 300× Past the Fit (Delphi) · predicting big models from small
Mixture of Experts Quantile Balancing: Validated at 32B-A5B (1e22 FLOPs) Scale · keeping MoE experts load balanced

Other models

Previously, we used Marin to train an 8B parameter model that outperformed Llama 3.1 8B on our base-model benchmark suite. You can see the training script or read the retrospective. We also trained Marin 32B.

Learning more & using Marin

The documentation for Marin is available on ReadTheDocs or in the docs/ folder.

To get started with Marin:

Install Marin.
Train a tiny language model using Marin.
See how to run a much larger DCLM 1B/1x experiment using Marin.
See a summary of the experiments we've run.
Join the Marin Discord to chat with the community.

Example

Marin experiments are defined as a set of steps that can depend on each other and are executed in a topological order, like a Makefile.

As a brief example of how you can use Marin, here is a complete script for training a tiny model on TinyStories. You can check out the full script for more details.

from fray.cluster import ResourceConfig
from levanter.optim import AdamConfig
from marin.execution.lazy import lower
from marin.execution.step_runner import StepRunner
from marin.experiment.data import tokenized
from marin.experiment.train import train_lm

from experiments.llama import llama_nano
from experiments.marin_tokenizer import marin_tokenizer

# 1. Tokenize the dataset as a lazy handle — nothing downloads yet.
tinystories_tokenized = tokenized(
    name="tokenized/tinystories",
    source="roneneldan/TinyStories",
    tokenizer=marin_tokenizer,
    sample_count=1000,  # cap at 1 000 samples per shard to keep the tutorial fast
)

# 2. Train the model — depends on the tokenized dataset above.
nano_tinystories_model = train_lm(
    name="checkpoints/marin-nano-tinystories",
    version="v1",
    model=llama_nano,
    optimizer=AdamConfig(learning_rate=6e-4, weight_decay=0.1),
    # Steps can depend on other steps: nano_tinystories_model depends on tinystories_tokenized
    datasets={tinystories_tokenized: 1.0},
    batch_size=4,
    seq_len=2048,
    num_train_steps=100,
    z_loss_weight=None,
    evals=None,  # no point evaluating such a tiny model
    resources=ResourceConfig.with_cpu(),
)

if __name__ == "__main__":
    StepRunner().run([lower(nano_tinystories_model)])

Here, we create two steps, one for tokenizing the dataset and one for training the model. The training step depends on the tokenized dataset step, so it will be executed after the tokenization step is completed.

With slight modifications, you can extend this to train a larger model on a larger dataset, a mixture of datasets, even scaling to very large GPU or TPU pods (or multislice TPUs!).

For Contributors

See CONTRIBUTING.md for project workflow.
See .agents/skills/ (also .claude/skills/) for loadable agent skills. For example, .agents/skills/add-dataset/ has a step-by-step guide to adding new datasets.

Core Contributors

Marin's core collaborators come from Stanford CRFM and Open Athena.

Supporters

Marin's research is made possible by the generous support of our partners.


_{for TRC accelerators}	_{for GPU clusters}

_{for supporting development}	_{for supporting development}