Soumendra's Blog

Large Language Models History

Large Language Models History

A brief history on the LLMs

Soumendra kumar sahoo's photo
Soumendra kumar sahoo
·

2 min read

What are Language Models?

A language model (LM) is a tool that guesses the next word in a given sequence of terms.

Evolution of Language Models

The development of LMs can be broadly classified into five stages.

  1. Rule-based LM

  2. Statistical LM

  3. Neural LM

  4. Pre-Trained LM

  5. Large Language Models (LLM)

Rule-based Language Models

  • Grammatical rules of a specific language were used to predict the next word in a sentence.

  • E.g., in English I will be followed by am not are, and They can be followed by have or are like these grammatical rules.

  • However, there are many exceptions, and handling all the language rules is tricky.

Statistical Language Models

  • In this method, a large set of texts was analyzed, and the word-level probability of a word after a bunch of words was determined statistically.

  • How many times does am appear after I that probability is compared with other words like are or is.

  • In an advanced SLM n-gram model, instead of finding probability from a previous single word, the last bi-gram (two words) and tri-grams (three terms) were used to find the possibility of the next word.

  • However, In English, a single word can have multiple meanings based on the context of the sentence. SLM can not able to determine the context of the sentence.

Neural Language Models

  • With Word2Vec (Word to Vector), these models calculate the probability of the following words by neural networks.

  • Example: RNN (Recurrent Neural Network), LSTM (Long Short Term Memory)

Pre-Trained Language Models

  • ELMo (Context-aware Word Embedding) and Self-Attention through Transformer architecture raised the performance bar of NLP tasks. Example: BERT and GPT-2

  • Models were trained with a large amount of text, and the context awareness increased.

Large Language Models (LLM)

  • There is a thin line between PLM and LLM.

  • Scaling model size and training data size of PLMs new emergent abilities of model discovered. Example: ChatGPT, LLaMA, Claude

  • LLM is different from PLM broadly in three ways:

    • Emergent abilities

    • Prompting/Conversational Interface

    • To attend the scale, Engineering and Research problems must be solved.

References

  1. Language Models in Plain English (oreilly.com)

  2. [2303.18223] A Survey of Large Language Models (arxiv.org)

large language modelsllm#languagemodellingnlp