but it is not succinctly summarized for consumption, and that is the function Wikipedia fulfills for most users
You may have noticed that this is precisely what I’ve disagreed on with two other users in this thread, who claimed that Wikipedia should contain as much information as possible. I agree with you, it is good to have an encyclopedia that summarises the most important, and directs towards further materials for further deeper study.
Again, the point I disagreed on with the other poster is simply on what’s the “biggest repository of human knowledge ever”. You seem to agree that the shadow libraries are bigger, and that settles it, we don’t have to disagree further. Being the biggest repository does not, of course, mean that it should be the only repository, that others are bad, or whatever else. Different repositories can fulfill different roles in the sharing of knowledge.
I can understand the pitfalls of allowing updates from the public, but I think it’s widely accepted that Wikipedia does more good than bad.
I don’t think I disagreed with the latter point, except when I mentioned the smaller borderline fascist wikis somewhere ITT. When it comes to English and some other large WPs I definitely don’t disagree.
Maybe you’re just supposing that we disagree on something when we really don’t, so you can’t lead the discussion to the point where we’d finally clearly disagree. I provided some criticism of Wikipedia, that doesn’t mean I think it’s bad wholesale. Wikipedia is complicated. I’ve spent quite a while editing it, as well as using traditional encyclopedias. I just negatively reacted to some comments in this thread, the harsh dismissal of traditional encyclopedias, and the somewhat overrated view of Wikipedia.
This was mostly noticed already in the earliest days of LLM spreading (I think several months since GPT went public), when Russian users noticed it has much worse output in Russian than English. The reason was simply that each letter was treated as a whole distinct token in Russian, leading to more difficulties in outputing words without typos. It is logical that this approach would be more resource-expensive and prone to errors, if the AI has to select each and every letter one by one instead of immediately jumping to selecting words and forming the whole sentence.
English is a fairly analytic language, words barely change, and subtleties in meaning and relationships are expressed with other words. Many other languages encode such information inside the word by various ways of modifying the same base form (inflectional languages). The same way English forms plurals by adding -s, so can other languages add a whole bunch of possible endings to express e.g. that the object is used as a tool, or that it’s the recipient of an action. To analyse and tokenise all that stuff would require decades of manual work. Alternatively I would assume there are some automatised tools that would generally correctly split the words into tokens, at least for the major inflectional languages, such as Russian. But most likely they just split it down to random syllables or even individual letters. Indeed, I just tested Russian, Polish and Croatian in their linked tokenizer, and the results are mostly syllabic or letter-based or just sliced into random sections that don’t correspond to anything linguistically relevant. Not a trace of morphemic analysis (morphemes are meaningful units, e.g. level-head-ed-ness, this style of slicing words down into pieces would probably be the ideal approach here). [Edit: holy shit, it actually seems to treat almost every capital letter as a distinct token.]
Or, to put it most simply, AI bullshits and brute-forces through languages with complicated words. And it’s been known for years. Nice to see additional confirmation, of course, and to learn that they still haven’t improved on it.