Very interesting, now I really wonder about Spanish, and Mandarin Chinese. There is so much written literature in Spanish that I would guess that much of it has been used to train models. For Chinese is a similar case but it has the interesting side effect to be a very different language than English. (English has germanic roots, but most of its words are latin based so not so far away from Spanish).
Finnish is in a totally different language family than Swedish-Norwegian-Danish. However Viking performs the same result. I am wondering the outcome when using Viking for Hungarian, as it is a near relative to Finnish.
Finnish and Hungarian split apart from their common ancestor in approximate 2200 BC, so they are not very near relatives, just relatives. In any event, over the millennia they lost much of their common vocabulary and were exposed to different contact influences, so performance on one doesn't make a big difference over other languages on its performance on the other.
Viking was trained on Finnish as well, so that's why it gets good results.
I ran a quick check on Hungarian, Viking didn't perform well at all(about 2x the tokens of English).
Would have been interesting to include German and Dutch. The German training corpus must be much bigger while the language is roughly the same "distance" from English as other Germanic languages except Dutch. Dutch is most similar to English, but likely has a smaller training corpus than German.
Very interesting, now I really wonder about Spanish, and Mandarin Chinese. There is so much written literature in Spanish that I would guess that much of it has been used to train models. For Chinese is a similar case but it has the interesting side effect to be a very different language than English. (English has germanic roots, but most of its words are latin based so not so far away from Spanish).
Finnish is in a totally different language family than Swedish-Norwegian-Danish. However Viking performs the same result. I am wondering the outcome when using Viking for Hungarian, as it is a near relative to Finnish.
Finnish and Hungarian split apart from their common ancestor in approximate 2200 BC, so they are not very near relatives, just relatives. In any event, over the millennia they lost much of their common vocabulary and were exposed to different contact influences, so performance on one doesn't make a big difference over other languages on its performance on the other.
Viking was trained on Finnish as well, so that's why it gets good results. I ran a quick check on Hungarian, Viking didn't perform well at all(about 2x the tokens of English).
Would have been interesting to include German and Dutch. The German training corpus must be much bigger while the language is roughly the same "distance" from English as other Germanic languages except Dutch. Dutch is most similar to English, but likely has a smaller training corpus than German.
I’d be curious to see this ran through cohere’s tokenizer - they seem to be the largest lab with a focus on multilingualism.