HDFS, MapReduce, Spark RDDs and DataFrames, lazy evaluation, partitioning, shuffles and Kafka streaming basics.
40 cards · basic cards · AI-written, checked twice. Edit anything.
- What is HDFS?
Show answer
Distributed file system designed for storing large files across clusters with high fault tolerance- What is the NameNode in HDFS?
Show answer
Master node that manages the file system namespace and controls file access- What is a DataNode in HDFS?
Show answer
Slave node that stores data blocks and performs block creation, deletion, and replication operations- What is block replication in HDFS?
Show answer
Storing copies of data blocks on multiple nodes for redundancy and fault tolerance- What is the default replication factor in HDFS?
Show answer
3- How does HDFS achieve fault tolerance?
Show answer
Through block replication and automatic replication of blocks from failed nodes to healthy ones- What is rack awareness in HDFS?
Show answer
The ability to optimize replica placement by understanding cluster topology and node locations- What is MapReduce?
Show answer
Programming model for distributed processing that divides jobs into map and reduce phases- What does the Map function do in MapReduce?
Show answer
Takes input data and produces intermediate key-value pairs- What does the Reduce function do in MapReduce?
Show answer
Aggregates values associated with the same key from the mapper output- What is the Shuffle and Sort phase in MapReduce?
Show answer
Process that sorts mapper output by key and groups values with identical keys for reducers- What is data locality in MapReduce?
Show answer
Principle of processing data on the node where it is stored rather than moving data across the network- What is a combiner in MapReduce?
Show answer
Mini-reducer that runs on mapper output to reduce data transferred to reducers- What are InputFormats in MapReduce?
Show answer
Specify how input data is split and read, determining the number of mappers and data distribution- What is an RDD in Spark?
Show answer
Immutable, distributed, fault-tolerant collection of objects that can be processed in parallel
FlashKeepers