How are keys and values presented and passed to the reducers during a standard sort and shuffle
phase of MapReduce?
A.
Keys are presented to reducer in sorted order; values for a given key are not sorted.
B.
Keys are presented to reducer in sorted order; values for a given key are sorted in ascending
order.
C.
Keys are presented to a reducer in random order; values for a given key are not sorted.
D.
Keys are presented to a reducer in random order; values for a given key are sorted in
ascending order.
Explanation:
Reducer has 3 primary phases:
1. Shuffle
The Reducer copies the sorted output from each Mapper using HTTP across the network.
2. Sort
The framework merge sorts Reducer inputs by keys (since different Mappers may have output thesame key).
The shuffle and sort phases occur simultaneously i.e. while outputs are being fetched they are
merged.
SecondarySort
To achieve a secondary sort on the values returned by the value iterator, the application should
extend the key with the secondary key and define a grouping comparator. The keys will be sorted
using the entire key, but will be grouped using the grouping comparator to decide which keys and
values are sent in the same call to reduce.
3. Reduce
In this phase the reduce(Object, Iterable, Context) method is called for each <key, (collection of
values)> in the sorted inputs.
The output of the reduce task is typically written to a RecordWriter via
TaskInputOutputContext.write(Object, Object).
The output of the Reducer is not re-sorted.
Reference: org.apache.hadoop.mapreduce, Class
Reducer<KEYIN,VALUEIN,KEYOUT,VALUEOUT>
D
0
0
A
0
0
A. Each of the mappers partition their intermediate data into as many reducers and than sorts on keys for each of the partitions. Each reducer receives intermediate data from as many mappers, and than sorts on key for all the partitions for a local total sort within the reducer. To achieve global total sort on the key, you have to write a custom partitioner than ensures that all keys for partition1 < all keys for partion2 < all keys for partionN
0
0
A.
The keys are sorted but the values for the Keys are not sorted.
0
0
A
0
0