Large Language Models – Engineering AI Agents

Author

Pantelis Monogioudis

Attention in RNN-based NMT

When you hear the sentence “the soccer ball is on the field,” you don’t assign the same importance to all 7 words. You primarily take note of the words “ball” “on,” and “field…

Introduction to NLP Pipelines

In this chapter, we will introduce the topic of processing with neural architectures language in general. This includes natural language, code etc.

Introduction to Recurrent Neural Networks (RNN)

Data streams are everywhere in our lives. Weather station sensor data arrive in streams indexed by time, financial trading data and obviously reading comprehension - one can…

Language Models

These notes heavily borrowing from the CS224N set of notes on Language Models.

Multi-head self-attention

Earlier we have seen examples with the token bear being in multiple grammatical patterns that also influence its meaning. To capture such multiplicity we can use multiple…

RNN-based Neural Machine Translation

These notes heavily borrowing from the CS229N 2019 set of notes on NMT.

Simple RNN

The simple RNN architecture with just a single layer of neurons that receive the input \(\mathbf{x}\) is shown below.

Single-head self-attention

In the simple attention mechanism, the attention weights are computed deterministically from the input context. We call the combination of context-free embedding (eg…

The Long Short-Term Memory (LSTM) Architecture

In the simple RNN we have seen the problem of explodig or vanishing gradients when the span of back-propagation is large (large \(\tau\)). Using the conceptual IIR filter…

Transformers and Self-Attention

For the explanation of decoder-based architectures such as those used by GPT, please see the repo https://github.com/pantelis/femtotransformers and the embedded comments…

Word2Vec Embeddings

In the so called classical NLP, words were treated as atomic symbols, e.g. hotel, conference, walk and they were represented with on-hot encoded (sparse) vectors e.g.