AI & Machine Learning Roadmap

• Ambiguity: "I saw the bank" (river bank or financial bank?).
• Context changes meaning across sentences.
• Human language is diverse, informal, and multilingual.

• Documents are collections of sentences.
• Sentences are sequences of tokens (words or subwords).
• The first step is turning raw text into countable units.

• Tokenization splits text into tokens (words, subwords, characters).
• The vocabulary is the set of unique tokens.
• Each token is then mapped to an integer ID.

• Lowercasing unifies "The" and "the".
• Punctuation removal reduces noise.
• Stop-word removal drops very common words ("the", "is") when they add little signal.

• Stemming chops word endings ("running" → "run") with simple rules.
• Lemmatization reduces a word to its dictionary form using context.
• Both reduce vocabulary size so related forms match.

• An n-gram is a sequence of n tokens (unigram = one word, bigram = two).
• Word frequency counts how often each token appears.
• Frequency is the basis of Bag-of-Words.

• Bag-of-Words represents a document as a count of each vocabulary word.
• It ignores word order entirely.
• Limitation: "dog bites man" and "man bites dog" become the same vector.

• TF (term frequency) counts how often a term appears in a document.
• IDF (inverse document frequency) down-weights terms that appear in many documents.
• TF-IDF highlights words that are frequent in one document but rare overall.

• Text classification assigns a label to a document.
• Sentiment analysis is classification into positive/negative/neutral.
• Named Entity Recognition identifies people, places, and organizations in text.

• A language model predicts the next token given the previous tokens.
• This is the foundation of both text generation and many representations.
• Training on next-token prediction is how large models learn structure.

• An embedding is a dense vector representing a word in a continuous space.
• Similar words have similar vectors ("king" near "queen").
• Embeddings capture relationships that counts cannot.

• A static embedding gives a word one vector regardless of context.
• But "bank" means different things in different sentences.
• Contextual representations compute a vector per word in its context.

• Recurrent networks process sequences step by step, carrying a hidden state.
• LSTMs add gates to remember information over long distances.
• They were the standard for sequences before Transformers.

Attention lets a model focus on the relevant parts of a sequence when computing each representation. In "the animal didn’t cross the street because it was too tired," the word "it" should attend to "animal."

• Each token produces a query, key, and value.
• Attention scores compare queries against keys to decide what to attend to.
• Values are weighted by those scores to form the output.

• Self-attention relates each token to every other token in the same sequence.
• Transformers are built entirely from self-attention and feed-forward layers.
• An encoder reads input; a decoder generates output.

This is a conceptual overview; the full mechanics are covered in the Generative AI & LLMs node.

• Ambiguity: the same words can mean different things.
• Bias: models learn statistical patterns, including harmful ones.
• Hallucination: models can produce fluent but false text.
• Multilingual systems add the challenge of many languages and scripts.

• Assuming tokenization equals understanding.
• Assuming word frequency captures meaning.
• Confusing embeddings with dictionaries.
• Misunderstanding attention as simply "looking at important words".