If you find the content useful and wish to support our platform’s development, you can contribute any amount toward our production costs. Scan the UPI QR code for payment within India. Or use the Ko-fi link to process a secure payment via PayPal.
Deep Learning
31 Terms
A–Z Linked
Deep Learning
Concepts related to neural networks, representation learning, and advanced AI model architectures.
Select a term below or scroll through the full category.
- Activation Function
- Artificial Neural Network
- Attention Mechanism
- Backpropagation
- Batch
- Convolutional Neural Network
- Deep Learning
- Embedding
- Emergent Capability
- Epoch
- Feed-Forward Network
- Forward Pass
- Gradient Descent
- Hidden Layer
- Input Layer
- Layer
- Layer Normalisation
- Logit
- Mini-Batch
- Mixture of Experts
- Multi-Head Attention
- Neuron
- Output Layer
- Positional Encoding
- Quantisation
- Recurrent Neural Network
- Representation Engineering
- Residual Connection
- Softmax
- Transformer
- Weight
Activation Function
📘 Definition
An activation function determines whether a neuron should pass information to the next layer and introduces non-linearity into the network.
🪄 Simple Explanation
It decides how a neuron responds to its inputs.
🔑 Why It Matters
Without activation functions, deep learning would be much less powerful.
💡 Example
ReLU is a commonly used activation function.
See Also
Artificial Neural Network
📘 Definition
An artificial neural network is a computational model inspired by the structure of the human brain. It consists of connected nodes that learn patterns from data.
🪄 Simple Explanation
A neural network learns by connecting many simple processing units.
🔑 Why It Matters
Neural networks are the foundation of deep learning.
💡 Example
A handwriting recognition system can be built using an artificial neural network.
See Also
Attention Mechanism
📘 Definition
An attention mechanism enables a neural network to focus on the most relevant parts of the input when making predictions or generating content.
🪄 Simple Explanation
Attention helps AI concentrate on the most important information.
🔑 Why It Matters
It greatly improves language understanding and generation.
💡 Example
While translating a sentence, the model focuses on the words that matter most.
See Also
Backpropagation
📘 Definition
Backpropagation is the learning algorithm that updates neural network weights by sending prediction errors backwards through the network.
🪄 Simple Explanation
Backpropagation helps a network learn from its mistakes.
🔑 Why It Matters
It is fundamental to training deep neural networks.
💡 Example
After an incorrect prediction, the network adjusts its weights using backpropagation.
See Also
Batch
📘 Definition
A batch is a subset of the training data processed together before model parameters are updated.
🪄 Simple Explanation
A batch is a small group of training examples processed at one time.
🔑 Why It Matters
Batches improve training efficiency.
💡 Example
Instead of using all images at once, a model trains on batches of 64 images.
See Also
Convolutional Neural Network
📘 Definition
A Convolutional Neural Network (CNN) is a deep learning architecture designed especially for analysing images and visual data.
🪄 Simple Explanation
CNNs are neural networks specialised for images.
🔑 Why It Matters
CNNs power many computer vision applications.
💡 Example
Image classification and medical image analysis commonly use CNNs.
Deep Learning
📘 Definition
Deep learning is a branch of machine learning that uses artificial neural networks with many layers to learn complex patterns from large amounts of data.
🪄 Simple Explanation
Deep learning uses many layers of learning to solve difficult problems.
🔑 Why It Matters
It powers many of today's most advanced AI applications.
💡 Example
Image recognition and modern language models rely heavily on deep learning.
See Also
- Machine Learning
- Neural Network
- Large Language Model
Embedding
📘 Definition
An embedding is a numerical representation of text, images, or other data that captures meaning by placing similar concepts close together in a mathematical space.
🪄 Simple Explanation
Embeddings convert meaning into numbers.
🔑 Why It Matters
They help AI compare information based on meaning rather than exact wording.
💡 Example
A search engine finds similar documents by comparing embeddings.
See Also
Emergent Capability
📘 Definition
An emergent capability is a behaviour or skill that appears as an AI model is scaled or trained more extensively, even though that capability was not explicitly programmed as a separate feature.
🪄 Simple Explanation
An emergent capability is a skill that appears as a model becomes larger or more capable without being directly programmed into it.
🔑 Why It Matters
Emergence helps researchers study how capabilities change with scale and why the behaviour of increasingly powerful AI systems may not always be straightforward to predict.
💡 Example
A larger language model may become noticeably better at a task that smaller versions handled poorly despite sharing a similar general training objective.
Continue Learning
Epoch
📘 Definition
An epoch is one complete pass through the entire training dataset during model training.
🪄 Simple Explanation
An epoch means the model has seen every training example once.
🔑 Why It Matters
Multiple epochs are usually needed for effective learning.
💡 Example
A model may train for 50 epochs before reaching good performance.
See Also
Feed-Forward Network
📘 Definition
A feed-forward network is a neural network in which information moves from input towards output without recurrent loops. Transformer blocks use feed-forward networks to transform token representations.
🪄 Simple Explanation
A feed-forward network transforms information as it moves forward through neural network layers.
🔑 Why It Matters
Feed-forward networks are fundamental building blocks of deep learning and perform a large share of the transformation inside transformer models.
💡 Example
After attention relates tokens to one another, a feed-forward network further transforms each resulting token representation.
See Also
- Neural Network
- Layer
- Transformer
Forward Pass
📘 Definition
A forward pass is the process in which data moves through a neural network from the input layer to the output layer to produce a prediction.
🪄 Simple Explanation
A forward pass is how a network generates an answer.
🔑 Why It Matters
Every prediction begins with a forward pass.
💡 Example
An image is passed through all network layers before a label is predicted.
See Also
Gradient Descent
📘 Definition
Gradient descent is an optimisation algorithm that gradually reduces prediction errors by adjusting model parameters in the direction of lower loss.
🪄 Simple Explanation
It is a method for improving a model step by step.
🔑 Why It Matters
Most deep learning models rely on gradient descent.
💡 Example
Training repeatedly updates parameters until errors become smaller.
See Also
Input Layer
📘 Definition
The input layer is the first layer of a neural network that receives raw data before it is processed by hidden layers.
🪄 Simple Explanation
The input layer accepts information for the network.
🔑 Why It Matters
It is the starting point for every neural network.
💡 Example
Pixels from an image enter a neural network through the input layer.
See Also
Layer
📘 Definition
A layer is a group of neurons that processes information before passing it to the next stage of a neural network.
🪄 Simple Explanation
Layers gradually transform raw data into meaningful patterns.
🔑 Why It Matters
Multiple layers enable deep learning models to solve difficult problems.
💡 Example
Early layers detect edges while deeper layers recognise complete objects in images.
See Also
Layer Normalisation
📘 Definition
Layer normalisation is a technique that normalises activations within a neural network layer to help keep values well-scaled and training stable.
🪄 Simple Explanation
Layer normalisation keeps internal neural network values in a more stable range while the model learns.
🔑 Why It Matters
It improves training stability and is an important component of transformer architectures, particularly when many layers are stacked together.
💡 Example
Transformer blocks use layer normalisation alongside attention, feed-forward processing, and residual connections.
See Also
Logit
📘 Definition
A logit is a raw numerical score produced by a model before it is converted into a probability. In language models, a logit is typically produced for each possible next token.
🪄 Simple Explanation
A logit is a model's raw score for a possible output before probabilities are calculated.
🔑 Why It Matters
Logits form the bridge between a neural network's internal calculations and the probabilities used to choose classifications or generated tokens.
💡 Example
An LLM produces logits for many possible next tokens before those scores are converted into probabilities.
See Also
- Probability
- Token
- Prediction
Mini-Batch
📘 Definition
A mini-batch is a small portion of the training dataset used during each update of a learning algorithm.
🪄 Simple Explanation
Mini-batches balance speed and learning quality.
🔑 Why It Matters
They are widely used in modern deep learning.
💡 Example
A model may process 32 examples in each mini-batch.
Mixture of Experts
📘 Definition
Mixture of Experts (MoE) is a model architecture containing multiple specialised sub-networks, or experts, with a routing mechanism that selects which experts should process a particular input.
🪄 Simple Explanation
A Mixture of Experts model sends each input to only some of several specialised parts of the network.
🔑 Why It Matters
MoE architectures can increase model capacity without activating every parameter for every input, allowing some AI models to scale more efficiently.
💡 Example
For one token, a routing system may activate two expert networks while leaving the model's other experts inactive.
See Also
- Neural Network
- Model Architecture
- Large Language Model
Multi-Head Attention
📘 Definition
Multi-head attention is a transformer mechanism that runs several attention operations in parallel, allowing different attention heads to learn different relationships among tokens.
🪄 Simple Explanation
Multi-head attention lets a transformer examine several kinds of relationships in the same text at once.
🔑 Why It Matters
Different heads can focus on different patterns, helping transformers build richer contextual representations.
💡 Example
One attention head may focus on grammatical relationships while another tracks connections between distant words.
See Also
Continue Learning
Neuron
📘 Definition
A neuron is the basic computational unit within an artificial neural network. It receives inputs, performs calculations, and passes information to other neurons.
🪄 Simple Explanation
A neuron is a small processing unit inside a neural network.
🔑 Why It Matters
Millions of neurons work together to solve complex tasks.
💡 Example
Each neuron contributes a small part of an image recognition decision.
Output Layer
📘 Definition
The output layer is the final layer of a neural network that produces the model's prediction or decision.
🪄 Simple Explanation
The output layer provides the final answer.
🔑 Why It Matters
It converts learned patterns into usable predictions.
💡 Example
A digit recognition model outputs the predicted number through the output layer.
See Also
Positional Encoding
📘 Definition
Positional encoding is a method of adding information about token order to the representations processed by a transformer.
🪄 Simple Explanation
Positional encoding tells a transformer where each token appears in a sequence.
🔑 Why It Matters
Attention does not inherently know token order, so positional information helps transformers understand sequence structure and differences caused by word order.
💡 Example
Positional information helps distinguish 'the dog chased the cat' from 'the cat chased the dog'.
See Also
Continue Learning
Quantisation
📘 Definition
Quantisation is a model optimisation technique that represents weights or calculations using lower-precision numerical formats, reducing the memory and computing resources required to run a model.
🪄 Simple Explanation
Quantisation makes a model smaller and more efficient by storing and processing numbers with less precision.
🔑 Why It Matters
It can make large models cheaper and faster to deploy on servers, personal computers, and edge devices, although excessive reduction in precision can affect quality.
💡 Example
A model using 16-bit weights may be converted to 8-bit or 4-bit representations so that it requires less memory during inference.
Recurrent Neural Network
📘 Definition
A Recurrent Neural Network (RNN) is a neural network architecture designed to process sequential data by retaining information from previous inputs.
🪄 Simple Explanation
RNNs remember earlier information in a sequence.
🔑 Why It Matters
They were an important step in language and speech processing before transformers.
💡 Example
Speech recognition systems traditionally used RNNs.
See Also
Representation Engineering
📘 Definition
Representation engineering is an approach to studying and influencing the internal representations learned by neural networks, particularly activation patterns associated with concepts, behaviours, or model states.
🪄 Simple Explanation
Representation engineering examines and sometimes adjusts how concepts are represented inside a neural network.
🔑 Why It Matters
It offers researchers another way to investigate model behaviour and may help improve interpretability, control, and safety in complex neural networks.
💡 Example
Researchers may analyse activation patterns inside a language model to investigate how a particular concept is represented across its layers.
See Also
- Interpretability
- Neural Network
- Embedding
Continue Learning
Residual Connection
📘 Definition
A residual connection is a neural network design in which the input to a layer or block is added to its output, creating a direct path through part of the network.
🪄 Simple Explanation
A residual connection lets information bypass part of a deep network instead of being completely transformed at every step.
🔑 Why It Matters
Residual connections help information and gradients move through deep networks, making very deep architectures such as transformers easier to train.
💡 Example
A transformer block adds its original input back to the result of an attention operation before processing continues.
See Also
- Neural Network
- Transformer
- Backpropagation
Softmax
📘 Definition
Softmax is a mathematical function that converts a set of model scores into a probability distribution in which the probabilities add up to one.
🪄 Simple Explanation
Softmax turns raw model scores into probabilities that can be compared.
🔑 Why It Matters
It is widely used in classification and language generation because it allows a model to express how likely each possible output is.
💡 Example
An LLM can apply softmax to its next-token scores so that each candidate token receives a probability.
See Also
- Probability
- Prediction
- Token
Transformer
📘 Definition
A transformer is a deep learning architecture that processes information using attention mechanisms, enabling efficient handling of long sequences. It forms the foundation of modern large language models.
🪄 Simple Explanation
Transformers allow AI to understand relationships across long pieces of information.
🔑 Why It Matters
They power today's most capable language and multimodal models.
💡 Example
ChatGPT and many modern AI systems are based on transformers.
Weight
📘 Definition
A weight is a numerical value that determines the importance of a connection between two neurons in a neural network.
🪄 Simple Explanation
Weights control how strongly neurons influence each other.
🔑 Why It Matters
Learning mainly involves adjusting weights.
💡 Example
Training changes weights until predictions improve.