PrepAway - Latest Free Exam Questions & Answers

Where is intermediate data written to after being emitted from the Mapper’s map method?

You have just executed a MapReduce job. Where is intermediate data written to after being
emitted from the Mapper’s map method?

PrepAway - Latest Free Exam Questions & Answers

A.
Intermediate data in streamed across the network from Mapper to the Reduce and is never
written to disk.

B.
Into in-memory buffers on the TaskTracker node running the Mapper that spill over and are
written into HDFS.

C.
Into in-memory buffers that spill over to the local file system of the TaskTracker node running
the Mapper.

D.
Into in-memory buffers that spill over to the local file system (outside HDFS) of the TaskTracker
node running the Reducer

E.
Into in-memory buffers on the TaskTracker node running the Reducer that spill over and are
written into HDFS.

Explanation:
The mapper output (intermediate data) is stored on the Local file system (NOT
HDFS) of each individual mapper nodes. This is typically a temporary directory location which can

be setup in config by the hadoop administrator. The intermediate data is cleaned up after the
Hadoop Job completes.
Reference: 24 Interview Questions & Answers for Hadoop MapReduce developers, Where is the
Mapper Output (intermediate kay-value data) stored ?

8 Comments on “Where is intermediate data written to after being emitted from the Mapper’s map method?

  1. Pradeep says:

    The mapper output (intermediate data) is stored on the Local file system (NOT HDFS) of each individual mapper nodes. This is typically a temporary directory location which can be setup in config by the hadoop administrator. The intermediate data is cleaned up after the Hadoop Job completes.




    0



    0
  2. johney says:

    I believe that ‘Outside of HDFS’ means the inter-key will be writen on the local FS in the directory other than the HDFS’s directory. These two directories is setup by ‘dfs.data.dir’and ‘dfs.tmp.dir’ in HDFS-site.xml




    0



    0

Leave a Reply to dennis Cancel reply

Your email address will not be published. Required fields are marked *