This reminds me of how I learned about Zipf's law. 19 years ago, I spent the summer after my sophomore year in a research lab at UC Santa Cruz as part of an NSF-funded summer research program for undergrads. My project that summer was to investigate index design for the Ceph distributed file system in order to enable faster file search. This was a hot topic at the time, especially with Spotlight being introduced to Mac OS X Tiger, and we wanted to see how to enable similar functionality for a distributed file system. I learned about Zipf's law, which states that there is a power law distribution of word frequencies in many natural languages. I remember downloading the Project Gutenberg corpus and writing a Python script where I stemmed the words and computed the frequencies of the stemmed words. The resulting distribution confirmed Zipf's law.
Sounds about right. Similar experience learning German when I moved to Berlin from the UK (though less extreme, felt like 85% of words was 50% of sentences).
Human languages are massively redundant specifically to enable this kind of "fill the gaps" learning process.
Observe an unknown word used a few times in different contexts, and you'll get a very good bead on what it means.
It's the "natural" way of learning a language. No one gets every word explained to them in detail - most words you know in your own first language are the ones you picked up on the go.
I think this "structural redundancy" might be a big part of what enables LLMs to learn human languages "from nothing".
And in the case of Japanese, with words consisting of multiple Kanji you can even get more context of a word from the word without knowing the word. Down to, to a lesser extent, radicals too...
I didn’t include it in my analysis, but I’ve been thinking about “sentence coverage” vs “word coverage” for a number of years now. In an extreme case, you could imagine a transcript with only i+1/1T sentences and you’d be left with totally incomprehensible input.
It's the 500 most common words in that specific show rather than some broader corpus, so there's bound to be some words that happen to be unusually common or rare.
One thing that seem to be improving Japanese literacy across the world is Inmu gay porn meme contents. I believe it hasn't reached the US, but it is becoming eerily well known in East/Southeast Asian countries, including mainland China. It uses tiny set of vocabulary in highly repetitive manner, a la kid's picture books, but in the way relatable and addicting to social media users rather than e.g. portraying a day in the life of a cute crayon drawn caterpillar.
IMO, what that tells us is that a completely dumb repetition of small set of simple concepts to just set up the pathway can be an effective means to kickstart language acquisition process. Most language learning materials attempt to establish a compact set of rules to be expanded with a long tail of modular add-ons, even many that aims to create an alternative and less stressful approaches like Duolingo. The Inmu dictionary has less than 500 known expression, all unusable in regular communications and inappropriate in most settings, and the way it is used is simplistic, childish, disrespectful, and dumb. And I think that might be how "start with picture books" advice look like if rebuilt from first principles.
Ironic that it holds the first place by understandable lines. The vocabulary isn't the complex thing - it's the content itself that eludes understanding.
Surreal and dreamlike - almost as if inviting you to connect the dots, but not giving you the labels, not giving you enough dots or an obvious enough outline to connect them confidently.
Observe an unknown word used a few times in different contexts, and you'll get a very good bead on what it means.
It's the "natural" way of learning a language. No one gets every word explained to them in detail - most words you know in your own first language are the ones you picked up on the go.
I think this "structural redundancy" might be a big part of what enables LLMs to learn human languages "from nothing".
>What could make this wrong: >1. "Understood" means you know every word. It ignores grammar, kanji readings, speaking speed and context.
I didn’t include it in my analysis, but I’ve been thinking about “sentence coverage” vs “word coverage” for a number of years now. In an extreme case, you could imagine a transcript with only i+1/1T sentences and you’d be left with totally incomprehensible input.
You blurred yakusoku, everybody knows that, but houshuu is much rarer word, should have blurred that
IMO, what that tells us is that a completely dumb repetition of small set of simple concepts to just set up the pathway can be an effective means to kickstart language acquisition process. Most language learning materials attempt to establish a compact set of rules to be expanded with a long tail of modular add-ons, even many that aims to create an alternative and less stressful approaches like Duolingo. The Inmu dictionary has less than 500 known expression, all unusable in regular communications and inappropriate in most settings, and the way it is used is simplistic, childish, disrespectful, and dumb. And I think that might be how "start with picture books" advice look like if rebuilt from first principles.
This person is visiting different learning sites then I do.
Also, I dont need to understand every word in sentence to understand it. Especially when coupled with visual show.
I'm pretty sure many of the lines are _not understandable by intent_, even.
"Present day... present time... ahahaha!"
Ironic that it holds the first place by understandable lines. The vocabulary isn't the complex thing - it's the content itself that eludes understanding.
Surreal and dreamlike - almost as if inviting you to connect the dots, but not giving you the labels, not giving you enough dots or an obvious enough outline to connect them confidently.