<aside> ☝

List of notes for this specialization + Lecture notes & Repository & Quizzes + Home page on Coursera. Read this note alongside the lecture notes—some points aren't mentioned here as they're already covered in the lecture notes.

</aside>

Course 3 overview

• Storage and queries are more complex than they appear.
• Various decisions in data storage impact performance and efficiency.
• Storage solution considerations: data type, data size, data format, access and update pattern.
• Storage hierarchy:
  • Management system: Organizes data in the raw components and allows you to interact with the stored data
  • OLTP (Online Transactional Processing Systems): focus on performing read and write queries with low latency. ← row oriented storage is more suitable
  • OLAP systems (Online Analytical Processing Systems): Focus on applying analytical activities on data (e.g. aggregation, summarization) ← column oriented storage is more suitable
• Plan:
  • Week 1: Trade off between storage cost and performance.
    • Cloud storage paradigms (block, object and file storage)
    • Data storage in databases
      • Row vs column-oriented databases
      • Graph and vector databases
    • Characteristics of physical components
    • Serialization and compression
  • Week 2: How to choose the appropriate abstractions for storing your data.
  • Week 3: Queries
    • How queries work
    • How different storage solutions affect query performance
    • Techniques for improving query performance

Data Storage Deep Dive

Storage Raw Ingredients - Physical Components of Data Storage

• Persistent Storage Medium: HDD, SSD.
• Volatile Memory: RAM, CPU cache.

Storage Raw Ingredients - Processes Required for Data Storage

• Serialization: Data need to be serialized before saving in storage and need to be de-serialized to be read.

  

• Serialization formats

  

• Compression

  

  

  Compression ratio = compressed files/original uncompressed files.

Cloud Storage Options: Block, Object and File storage

• File Storage:

  • Organizes data in a hierarchical directory structure.
  • Suitable for centralized access and easy sharing among users.
  • Lower read/write performance due to tracking file hierarchy.
  • Amazon Elastic File System (EFS)

  

• Block Storage:

  • Stores data in fixed-size blocks.
  • High performance and low latency for frequent read/write operations.
  • Ideal for transactional workloads and virtual machine storage.
  • Amazon Elastic Block Store (EBS)

  

• Object Storage:

  • Stores data as immutable objects in a flat structure.
  • Highly scalable, supporting petabytes of data.
  • Best for analytical workloads, data lakes, and large unstructured data storage.
  • Amazon S3