Scope

Define and develop the tokenization used by the model:

• the tokenization algorithm itself
• the tokenization in the context of multilinguality
• implement it and coordinate with the dataset and modeling working groups to prepare the data for training

Resources - onboarding - documentation

No ressource at the moment

Current members

M Saiful Bari, Anthony Moi, Victor Sanh, Samson Tan, Francois Yvon, Wietse de Vries, Mohamed Ahmed, Wilson Lee, Sandip Patil, Rico Sennrich, Shiyue Zhang, Maxime De Bruyn, Vassilina NIkoulina, Zaid Alyafeai, Nick Doiron, Ben Peters, Chenglei Si, Mike Tian-Jian Jiang, Kamalkumar Rathinasamy, Kaustubh Dhole, Benoît Sagot, Roman Castagné, Thomas Wang, Uğur Ünal, Md Rabiul Awal, Tasnim Mohiuddin, Rachel Bawden, Elizabeth Keleshian, Manan Dey, Shannon Shen, Sadid A. Hasan, Arun Raja, Priyank Soni, Bjarke Felbo, Sabrina Mielke, Shamsuddeen Muhammad, Taewoon Kim, Zhiqing Sun, Elizabeth Salesky

Current chair(s)

Benoît Sagot, Samson Tan

Current status

Kicked off the discussion