Introduction to Alternative Databases and Data Warehouses
Subtopic A3.4 covers two closely related areas that form the backbone of modern large-scale data management:
- Alternative database models , non-relational (NoSQL) databases that address limitations of traditional relational systems when dealing with unstructured, high-volume, or highly connected data.
- Data warehouses , large-scale analytical repositories that consolidate historical data from multiple sources to support business intelligence (BI), including OLAP and data mining.
Understanding why these alternatives emerged requires recognising the limitations of traditional relational databases (RDBMS) when confronted with modern data challenges.
The Rise of Big Data
Modern organisations generate data at unprecedented scale, speed, and variety , often summarised as the Three Vs:
- Volume , petabytes of data generated daily (social media posts, sensor readings, transaction logs).
- Velocity , data arriving in real time or near-real time (financial trades, IoT streams, user clicks).
- Variety , data in structured (tables), semi-structured (JSON, XML), and unstructured (images, text, video) formats.
Traditional relational databases, while excellent for structured transactional data, struggle with all three Vs simultaneously. This created demand for alternative approaches.
Data Structures: A Motivating Framework
| Type | Description | Examples |
|---|---|---|
| Structured | Fixed schema, rows and columns | SQL tables, spreadsheets |
| Semi-structured | Flexible schema, self-describing | JSON, XML, CSV |
| Unstructured | No predefined schema | Images, video, emails, social posts |
Traditional RDBMS handle structured data well but are ill-suited to semi-structured and unstructured data at scale. NoSQL databases were designed to fill this gap.
Alternative Databases: NoSQL Models
NoSQL ("Not Only SQL") databases are non-relational database systems designed for flexibility, horizontal scalability, and performance with large volumes of varied data. Unlike relational databases, they do not require a fixed schema and do not always enforce ACID properties in the traditional sense.
NoSQL Database: A non-relational database management system that stores and retrieves data using models other than the tabular rows-and-columns structure of relational databases. NoSQL systems prioritise scalability, flexibility, and performance for specific data patterns over strict relational integrity.
There are four primary NoSQL database models, each optimised for different data structures and use cases:
1. Key-Value Stores
The simplest NoSQL model. Data is stored as pairs of unique keys and their associated values. The value can be anything , a string, number, JSON object, or binary blob. The database does not interpret the value; it simply stores and retrieves it by key.
- Strengths: Extremely fast lookups; highly scalable; simple to implement.
- Weaknesses: No querying on value contents; poor for complex relationships.
- Use cases: Session management, user preferences, shopping carts, caching.
- Examples: Redis, Amazon DynamoDB (in key-value mode), Memcached.
An e-commerce site stores each logged-in user's session data as:
Key: "session_abc123" → Value: {user_id: 9812, cart: [{item: "headphones", qty: 1}], last_active: "2024-03-15T14:22:00Z"}
Looking up a session by its key takes microseconds, regardless of how many millions of sessions exist.
2. Document Stores
Stores data as documents , typically in JSON or BSON format. Each document is self-describing and can have a different structure from other documents in the same collection. Related data is nested within the document rather than spread across linked tables.
- Strengths: Schema flexibility; natural fit for object-oriented data; rich query support on document contents.
- Weaknesses: Less efficient for highly relational data with many cross-document links.
- Use cases: Content management systems, user profiles, product catalogues, mobile app data.
- Examples: MongoDB, CouchDB, Firebase Firestore.
A product catalogue in a document store:
{
"product_id": "P4521",
"name": "Wireless Headphones",
"brand": "SoundMax",
"specs": {
"battery_hours": 30,
"connectivity": ["Bluetooth 5.0", "USB-C"]
},
"reviews": [
{"user": "alice", "rating": 5, "comment": "Excellent sound"},
{"user": "bob", "rating": 4, "comment": "Comfortable fit"}
]
}
Note that different products can have entirely different fields , a book document would have author and isbn fields, with no need to alter a shared schema.
3. Column-Family Stores (Wide-Column Stores)
Organizes data into rows and columns like a relational database, but with a critical difference: different rows can have different columns, and columns are grouped into column families. Data is stored and retrieved column-by-column rather than row-by-row, making analytical queries over specific columns extremely fast.
- Strengths: Exceptional performance for analytical read queries over large datasets; efficient compression; scales horizontally across many servers.
- Weaknesses: Complex data model; less intuitive for developers; poor for ad-hoc joins.
- Use cases: Time-series data, event logs, IoT sensor data, large-scale analytics.
- Examples: Apache Cassandra, Google Bigtable, Apache HBase.
A telecommunications company stores call records for billions of customers in a column-family store. To calculate the average call duration for a specific region in Q3, the database reads only the call_duration and region columns , skipping all other stored fields. In a row-based RDBMS, every full row would be loaded into memory first, making the same query far slower at this scale.
4. Graph Databases
Represents data as nodes (entities) and edges (relationships between entities), with both nodes and edges able to carry properties. Designed to efficiently traverse and query complex, interconnected relationship networks.
- Strengths: Native representation of relationships; highly efficient for traversal queries ("find all friends of friends"); intuitive for network data.
- Weaknesses: Not suited for aggregate analytics on large flat datasets; steeper learning curve.
- Use cases: Social networks, fraud detection networks, recommendation engines, knowledge graphs, supply chain mapping.
- Examples: Neo4j, Amazon Neptune, ArangoDB.
A social media platform uses a graph database where:
- Nodes: Users, Posts, Hashtags, Pages
- Edges: FOLLOWS, LIKED, POSTED, TAGGED_IN
The query "Find all users who follow the same accounts as User X and have also liked posts tagged #travel" , which would require multiple expensive JOIN operations in SQL , is a natural traversal in a graph database.
Comparison of NoSQL Models
| Model | Data Structure | Best For | Example Systems |
|---|---|---|---|
| Key-Value | Key → arbitrary value | Fast lookups, caching | Redis, DynamoDB |
| Document | JSON/BSON documents | Flexible schemas, content | MongoDB, Firestore |
| Column-Family | Column-oriented rows | Large-scale analytics, IoT | Cassandra, Bigtable |
| Graph | Nodes and edges | Relationships, networks | Neo4j, Neptune |
NoSQL databases typically sacrifice some relational guarantees (e.g., strict ACID transactions, enforced referential integrity) in exchange for scalability and flexibility. This trade-off , sometimes described as BASE (Basically Available, Soft state, Eventually consistent) vs. ACID , is a key conceptual distinction for HL students.