BigScience Reddit AMA on Thursday, March 24th, 2022

Come ask questions about everything on the new HuggingFace BigScience language model, data, the licenses, and the cluster!

We have started training a 176B parameter multilingual language model on the French supercomputer Jean Zay – literally out in the open. This is not only the first time a multilingual LLM (46 languages!) at this scale will be fully accessible to the ML research community, but the whole decision, engineering and training process is transparent and open. We'll be training for several months and the community can follow along, engage and ask questions, etc. For regular training updates follow @BigScienceLLM.

The model, compute and training

• 176B parameters – 70 layers, 112 attention heads
• 384 A100 80GB GPUs– on Jean Zay
• Checkpoint size: only the bf16 weights are 329GB, the full checkpoint with optimizer states is 2.3TB
• Training throughput: about 150 TFLOPs
• Estimated training time: 3-4 months (depending on throughput and unexpected events)

More info

• Model architecture and a blog post on decisions on architecture, size, shape, and pretraining duration
• Tensorboard during the training
• Details on the obstacles overcome during the preparation on the engineering side (instabilities, optimization of training throughput, many technical challenges and questions). For ongoing chronicles since the start of the final training see chronicles.

For the AMA we’re joined by

Model / Engineering

• Thomas Wolf (Hugging Face) /u/Thomjazz