Engineering AI Agents
BOOK
Foundations
Training Deep Networks
Perception
State Estimation
Large Language Models
Multimodal Reasoning
Planning
Markov Decision Processes
Reinforcement Learning
COURSES
Introduction to AI
AI for Robotics
Deep Learning for Computer Vision
DATA MINING - BEING PORTED
ABOUT ME
Large Language Models
Large Language Models
Building foundational models that understand and generate human language
Author
Pantelis Monogioudis
Large Language Models
Introduction to NLP Pipelines
Text Tokenization
Word2Vec Embeddings {#sec-word2vec}
Word2Vec from scratch
Word2Vec Tensorflow Tutorial
Introduction to Recurrent Neural Networks (RNN)
Simple RNN
Time Series Prediction using RNNs
The Long Short-Term Memory (LSTM) Architecture
Language Models
RNN Language Models
Example of an RNN Language Model
RNN-based Neural Machine Translation
Character-level recurrent sequence-to-sequence model
The BLEU Score
Attention in RNN-based NMT
Transformers and Self-Attention
Single-head self-attention
Multi-head self-attention
Positional Embeddings
LLM Inference
Categories
All
(22)
Introduction to Recurrent Neural Networks (RNN)
Data streams are everywhere in our lives. Weather station sensor data arrive in streams indexed by time, financial trading data and obviously reading comprehension - one can…
Simple RNN
The simple RNN architecture with just a single layer of neurons that receive the input
\(\mathbf{x}\)
is shown below.
The Long Short-Term Memory (LSTM) Architecture
In the simple RNN we have seen the problem of explodig or vanishing gradients when the span of back-propagation is large (large
\(\tau\)
). Using the conceptual IIR filter…
Introduction to NLP Pipelines
In this chapter, we will introduce the topic of processing with neural architectures language in general. This includes natural language, code etc.
Language Models
These notes heavily borrowing from the CS224N set of notes on Language Models.
RNN-based Neural Machine Translation
These notes heavily borrowing from the CS229N 2019 set of notes on NMT.
Attention in RNN-based NMT
When you hear the sentence “the soccer ball is on the field,” you don’t assign the same importance to all 7 words. You primarily take note of the words “
ball
” “
on
,” and “
field
…
Transformers and Self-Attention
For the explanation of decoder-based architectures such as those used by GPT, please see the repo https://github.com/pantelis/femtotransformers and the embedded comments…
Single-head self-attention
In the simple attention mechanism, the attention weights are computed
deterministically
from the input context. We call the combination of context-free embedding (eg…
Multi-head self-attention
Earlier we have seen examples with the token
bear
being in multiple grammatical patterns that also influence its meaning. To capture such multiplicity we can use multiple…
No matching items
