PrepAway - Latest Free Exam Questions & Answers

Determine how many Mappers will run?

Your cluster’s HDFS block size in 64MB. You have directory containing 100 plain text files, each of

which is 100MB in size. The InputFormat for your job is TextInputFormat. Determine how many
Mappers will run?

PrepAway - Latest Free Exam Questions & Answers

A.
64

B.
100

C.
200

D.
640

Explanation:
Each file would be split into two as the block size (64 MB) is less than the file size
(100 MB), so 200 mappers would be running.
Note:
If you’re not compressing the files then hadoop will process your large files (say 10G), with a
number of mappers related to the block size of the file.
Say your block size is 64M, then you will have ~160 mappers processing this 10G file (160*64 ~=
10G). Depending on how CPU intensive your mapper logic is, this might be an
acceptable blocks size, but if you find that your mappers are executing in sub minute times, then
you might want to increase the work done by each mapper (by increasing the block size to 128,
256, 512m – the actual size depends on how you intend to process the data).
Reference: http://stackoverflow.com/questions/11014493/hadoop-mapreduce-appropriate-inputfiles-size (first answer, second paragraph)

4 Comments on “Determine how many Mappers will run?


Leave a Reply

Your email address will not be published. Required fields are marked *