Recurrent Neural Network (RNN)
A neural network architecture designed for sequential data that maintains a hidden state updated at each time step, allowing it to process variable-length sequences like text, time series, and audio.
RNNs process sequences one element at a time, maintaining a hidden state that serves as a memory of what has been seen so far. At each step, the network combines the current input with the previous hidden state to produce an output and an updated hidden state. This recurrence allows RNNs to handle variable-length sequences and capture temporal dependencies.
The basic RNN suffers from vanishing gradients, making it difficult to learn long-range dependencies. LSTM (Long Short-Term Memory) and GRU (Gated Recurrent Unit) architectures solve this with gating mechanisms that control what information to remember and forget. These gated variants were the dominant sequence modeling approach before transformers.
Transformers have largely replaced RNNs for language tasks because their parallel processing enables faster training on GPUs and their attention mechanism handles long-range dependencies more effectively. However, RNNs and their variants still have niches: real-time streaming applications where processing must happen sequentially, edge devices with limited memory (RNNs have constant memory regardless of sequence length), and certain time-series forecasting tasks. Understanding RNNs also provides context for why transformers were such a breakthrough.
Related Terms
RAG (Retrieval-Augmented Generation)
A technique that grounds LLM responses in external data by retrieving relevant documents at query time and injecting them into the prompt context.
Embeddings
Dense vector representations of text, images, or other data that capture semantic meaning in a high-dimensional space, enabling similarity search and clustering.
Vector Database
A specialized database optimized for storing, indexing, and querying high-dimensional vector embeddings with sub-millisecond similarity search.
LLM (Large Language Model)
A neural network trained on massive text corpora that can generate, understand, and transform natural language for tasks like summarization, classification, and conversation.
Fine-Tuning
The process of further training a pre-trained LLM on a domain-specific dataset to specialize its behavior, style, or knowledge for a particular task.
Prompt Engineering
The practice of designing and iterating on LLM input instructions to reliably produce desired outputs for a specific task.