You want to count the number of occurrences for each unique word in the supplied input data.
You’ve decided to implement this by having your mapper tokenize each word and emit a literal
value 1, and then have your reducer increment a counter for each literal 1 it receives. After
successful implementing this, it occurs to you that you could optimize this by specifying a
combiner. Will you be able to reuse your existing Reduces as your combiner in this case and why
or why not?
A.
Yes, because the sum operation is both associative and commutative and the input and output
types to the reduce method match.
B.
No, because the sum operation in the reducer is incompatible with the operation of a Combiner.
C.
No, because the Reducer and Combiner are separate interfaces.
D.
No, because the Combiner is incompatible with a mapper which doesn’t use the same data
type for both the key and value.
E.
Yes, because Java is a polymorphic object-oriented language and thus reducer code can be
reused as a combiner.
Explanation:
Combiners are used to increase the efficiency of a MapReduce program. They are
used to aggregate intermediate map output locally on individual mapper outputs. Combiners can
help you reduce the amount of data that needs to be transferred across to the reducers. You can
use your reducer code as a combiner if the operation performed is commutative and associative.
The execution of combiner is not guaranteed, Hadoop may or may not execute a combiner. Also, if
required it may execute it more then 1 times. Therefore your MapReduce jobs should not depend
on the combiners execution.
Reference: 24 Interview Questions & Answers for Hadoop MapReduce developers, What are
combiners? When should I use a combiner in my MapReduce Job?
A
0
0
Should be “B”
In the reducer it uses “counter += 1” instead of “counter = sum(word)”. Combiner may be called multiple times, thus you cannot use “counter += 1”.
In other word, it’s not associative and commutative.
0
0
I choose A
0
0