It's a bit difficult to explain LLMs succinctly, but, in essence, they're
artificial neural networks. You can think of certain words as points, or neurons, in a very large network of interconnected concepts, like a brain. Because of this diffusion of information through the network by not just having these points individually, but also by having complex relationships between them through the weights of interconnection (similar to how neuronal connections and strength of those connections vary between cells). Like OpenAI said in their
GPT-2 paper, language has a kind of "natural sequential ordering." The relationship between a noun and a verb allows for a fairly good probabilistic estimate on which follows what. "See Spot run." This means that you can train these networks to act in probabilistic ways with language.
So, it's less that these models are acting as a kind of specific "search and response" when it comes to user queries. They don't need to query the net like Jeeves to pull an answer and then use their knowledge of just language to format the reply (though I believe some advancement is being made in tying them to online searching, pointed out earlier in the thread by someone.) It's not like the old analogy of a filing cabinet. It's closer to how a human mind operates, in the linkage of related concepts in hierarchical structures and meaning. The
language is the information. They trained these models on vast scrapes of the internet and then distilled that information into effectively weights of what word should follow the next. The models are still
very large in size, but not as much as if you had the entire corpus laid out in individual segments. It's because they're created through vectoring by way of literal
billions of
hyperparameters, with the
smallest local models people are using today consisting of ones that are 7 or 13 billion. Hell, GPT-2 came out in 2019 and had only
1.5B, GPT-3 came out in 2020 with 175B, and GPT-4 is rumored to have 1.76
Trillion. That's how quickly they've scaled up.
So, local models (ones you can download the weights and run on your hardware, often given additional training to
remove the moralizing aspects) are definitely local, but are similarly "repositories" like other models that you have to run through online APIs (like GPT-4 or Claude) due to their unavailability (and size). I wouldn't necessarily call it a repository of
all human knowledge or anything, but you can definitely expect things like Wikipedia or other online sources that are part of
Common Crawl to be a part of GPT/Claude/LLaMA/Whatever. Jailbreaking is basically just using a set of instructions as part of the input to override the
other instructions sent as input that say "Be nice to everyone and don't say bad words and tell the user 'no' if they want you to say naughty things." The ones operating these APIs can't really prevent this (though they can try to add certain phrases/words to an appended blacklist to result in spitting out refusals), since reading input text and then trying to predict what words should follow are the fundamental operating mechanism for these models.
