Skip to main content

Important Hadoop Configurtions

Updated 2 min readMarkdown
Part of seriesData Engineering
Important Hadoop Configurtions

In this article, we will learn about important Hadoop configuration files

hadoop-env.sh

Environment variables that are used in the scripts to run Hadoop

Exploring core-site.xml

All the configuration settings related to Hadoop core such as I/O settings that are common to HDFS and MapReduce.

Reference Link: https://hadoop.apache.org/docs/r2.6.2/hadoop-project-dist/hadoop-common/core-default.xml

Exploring hdfs-site.xml

  • Configuration settings for HDFS daemons, the namenode, the secondary namenode and the data nodes.

  • Configuring Replication Factor, Block Size, Directory Specific Details, Permission, Security Level

dfs.block.size

This property is used to change the block size from its default size i.e 128 MB

<property>
    <name>dfs.block.sizw</name>
    <value>134217728</value>
</property>

dfs.replication

This property is used to change the replication factor from its default.

<property>
    <name>dfs.replication</name>
    <value>3</value>
</property>

dfs.permissions

If true, enable permission checking in HDFS. If false permission checking turned off.

<property>
     <name>dfs.permissions</name>
     <value>false</value>
</property>

dfs.namenode.name.dir

This property determines where on the local files system the DFS NameNode should store the NameNode table.

<property>
    <name>dfs.namenode.name.dir</name>
    <value>/home/npntraining/dfs/namenode</value>
</property>

dfs.datanode.data.dir

This property determines where on the local files system the DFS DataNode should store the blocks#

<property>
    <name>dfs.datanode.data.dir</name>
    <value>/home/npntraining/dfs/namenode</value>
</property>

Exploring mapred-site.xml

Configuration settings for MapReduce daemons : the ResourceManager and the NodeManager

mapreduce.framework.name

The runtime framework for executing MapReduce jobs. Can be one of local, classic or YARN

<property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
</property>

mapreduce.map.memory.mb

The amount of memory to request from the scheduler for each map task.

<property>
    <name>mapreduce.map.memory.mb</name>
    <value>1024</value>
</property>

mapreduce.map.cpu.vcores

The number of virtual cores to request from the scheduler for each map task..

<property>
    <name>mapreduce.map.cpu.vcores</name>
    <value>1</value>
</property>

mapreduce.reduce.memory.mb

The amount of memory to request from the scheduler for each reduce task.

<property>
    <name>mapreduce.reduce.memory.mb</name>
    <value>1024</value>
</property>

mapreduce.reduce.cpu.vcores

The number of virtual cores to request from the scheduler for each reduce task.

<property>
    <name>mapreduce.reduce.cpu.vcores</name>
    <value>1</value>
</property>

Exploring yarn-site.xml

mapreduce.framework.name

<property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
</property>

masters

A list of machines (one per line) that each run a secondary namenode.

slaves

A list of machines (one per line) that each run a DataNode and a NodeManager.

...

Connect with me on LinkedIn, If you are looking for 1:1 mentorship for a career and interviews connect me on 𝐭𝐨𝐩𝐦𝐚𝐭𝐞.𝐢𝐨/𝐧𝐚𝐯𝐞𝐞𝐧𝐩𝐧

About the author

  • I am a Tech Enthusiast having 13+ years of experience in 𝐈𝐓 as a 𝐂𝐨𝐧𝐬𝐮𝐥𝐭𝐚𝐧𝐭, 𝐂𝐨𝐫𝐩𝐨𝐫𝐚𝐭𝐞 𝐓𝐫𝐚𝐢𝐧𝐞𝐫, 𝐌𝐞𝐧𝐭𝐨𝐫, with 12+ years in training and mentoring in 𝐒𝐨𝐟𝐭𝐰𝐚𝐫𝐞 𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫𝐢𝐧𝐠, 𝐃𝐚𝐭𝐚 𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫𝐢𝐧𝐠, 𝐓𝐞𝐬𝐭 𝐀𝐮𝐭𝐨𝐦𝐚𝐭𝐢𝐨𝐧 𝐚𝐧𝐝 𝐃𝐚𝐭𝐚 𝐒𝐜𝐢𝐞𝐧𝐜𝐞. I have 𝒕𝒓𝒂𝒊𝒏𝒆𝒅 𝒎𝒐𝒓𝒆 𝒕𝒉𝒂𝒏 10,000+ 𝑰𝑻 𝑷𝒓𝒐𝒇𝒆𝒔𝒔𝒊𝒐𝒏𝒂𝒍𝒔 and 𝒄𝒐𝒏𝒅𝒖𝒄𝒕𝒆𝒅 𝒎𝒐𝒓𝒆 𝒕𝒉𝒂𝒏 500+ 𝒕𝒓𝒂𝒊𝒏𝒊𝒏𝒈 𝒔𝒆𝒔𝒔𝒊𝒐𝒏𝒔 in the areas of 𝐒𝐨𝐟𝐭𝐰𝐚𝐫𝐞 𝐃𝐞𝐯𝐞𝐥𝐨𝐩𝐦𝐞𝐧𝐭, 𝐃𝐚𝐭𝐚 𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫𝐢𝐧𝐠, 𝐂𝐥𝐨𝐮𝐝, 𝐃𝐚𝐭𝐚 𝐀𝐧𝐚𝐥𝐲𝐬𝐢𝐬, 𝐃𝐚𝐭𝐚 𝐕𝐢𝐬𝐮𝐚𝐥𝐢𝐳𝐚𝐭𝐢𝐨𝐧𝐬, 𝐀𝐫𝐭𝐢𝐟𝐢𝐜𝐢𝐚𝐥 𝐈𝐧𝐭𝐞𝐥𝐥𝐢𝐠𝐞𝐧𝐜𝐞 𝐚𝐧𝐝 𝐌𝐚𝐜𝐡𝐢𝐧𝐞 𝐋𝐞𝐚𝐫𝐧𝐢𝐧𝐠. I am interested in 𝐰𝐫𝐢𝐭𝐢𝐧𝐠 𝐛𝐥𝐨𝐠𝐬, 𝐬𝐡𝐚𝐫𝐢𝐧𝐠 𝐭𝐞𝐜𝐡𝐧𝐢𝐜𝐚𝐥 𝐤𝐧𝐨𝐰𝐥𝐞𝐝𝐠𝐞, 𝐬𝐨𝐥𝐯𝐢𝐧𝐠 𝐭𝐞𝐜𝐡𝐧𝐢𝐜𝐚𝐥 𝐢𝐬𝐬𝐮𝐞𝐬, 𝐫𝐞𝐚𝐝𝐢𝐧𝐠 𝐚𝐧𝐝 𝐥𝐞𝐚𝐫𝐧𝐢𝐧𝐠 new subjects.

No comments yet. Be the first to comment.

More from this blog