
In this article, we will learn about important Hadoop configuration files
hadoop-env.sh
Environment variables that are used in the scripts to run Hadoop
Exploring core-site.xml
All the configuration settings related to Hadoop core such as I/O settings that are common to HDFS and MapReduce.
Reference Link: https://hadoop.apache.org/docs/r2.6.2/hadoop-project-dist/hadoop-common/core-default.xml
Exploring hdfs-site.xml
Configuration settings for HDFS daemons, the namenode, the secondary namenode and the data nodes.
Configuring Replication Factor, Block Size, Directory Specific Details, Permission, Security Level
dfs.block.size
This property is used to change the block size from its default size i.e 128 MB
<property>
<name>dfs.block.sizw</name>
<value>134217728</value>
</property>
dfs.replication
This property is used to change the replication factor from its default.
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
dfs.permissions
If true, enable permission checking in HDFS. If false permission checking turned off.
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
dfs.namenode.name.dir
This property determines where on the local files system the DFS NameNode should store the NameNode table.
<property>
<name>dfs.namenode.name.dir</name>
<value>/home/npntraining/dfs/namenode</value>
</property>
dfs.datanode.data.dir
This property determines where on the local files system the DFS DataNode should store the blocks#
<property>
<name>dfs.datanode.data.dir</name>
<value>/home/npntraining/dfs/namenode</value>
</property>
Exploring mapred-site.xml
Configuration settings for MapReduce daemons : the ResourceManager and the NodeManager
mapreduce.framework.name
The runtime framework for executing MapReduce jobs. Can be one of local, classic or YARN
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
mapreduce.map.memory.mb
The amount of memory to request from the scheduler for each map task.
<property>
<name>mapreduce.map.memory.mb</name>
<value>1024</value>
</property>
mapreduce.map.cpu.vcores
The number of virtual cores to request from the scheduler for each map task..
<property>
<name>mapreduce.map.cpu.vcores</name>
<value>1</value>
</property>
mapreduce.reduce.memory.mb
The amount of memory to request from the scheduler for each reduce task.
<property>
<name>mapreduce.reduce.memory.mb</name>
<value>1024</value>
</property>
mapreduce.reduce.cpu.vcores
The number of virtual cores to request from the scheduler for each reduce task.
<property>
<name>mapreduce.reduce.cpu.vcores</name>
<value>1</value>
</property>
Exploring yarn-site.xml
mapreduce.framework.name
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
masters
A list of machines (one per line) that each run a secondary namenode.
slaves
A list of machines (one per line) that each run a DataNode and a NodeManager.
...
Connect with me on LinkedIn, If you are looking for 1:1 mentorship for a career and interviews connect me on 𝐭𝐨𝐩𝐦𝐚𝐭𝐞.𝐢𝐨/𝐧𝐚𝐯𝐞𝐞𝐧𝐩𝐧
About the author
I am a Tech Enthusiast having 13+ years of experience in 𝐈𝐓 as a 𝐂𝐨𝐧𝐬𝐮𝐥𝐭𝐚𝐧𝐭, 𝐂𝐨𝐫𝐩𝐨𝐫𝐚𝐭𝐞 𝐓𝐫𝐚𝐢𝐧𝐞𝐫, 𝐌𝐞𝐧𝐭𝐨𝐫, with 12+ years in training and mentoring in 𝐒𝐨𝐟𝐭𝐰𝐚𝐫𝐞 𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫𝐢𝐧𝐠, 𝐃𝐚𝐭𝐚 𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫𝐢𝐧𝐠, 𝐓𝐞𝐬𝐭 𝐀𝐮𝐭𝐨𝐦𝐚𝐭𝐢𝐨𝐧 𝐚𝐧𝐝 𝐃𝐚𝐭𝐚 𝐒𝐜𝐢𝐞𝐧𝐜𝐞. I have 𝒕𝒓𝒂𝒊𝒏𝒆𝒅 𝒎𝒐𝒓𝒆 𝒕𝒉𝒂𝒏 10,000+ 𝑰𝑻 𝑷𝒓𝒐𝒇𝒆𝒔𝒔𝒊𝒐𝒏𝒂𝒍𝒔 and 𝒄𝒐𝒏𝒅𝒖𝒄𝒕𝒆𝒅 𝒎𝒐𝒓𝒆 𝒕𝒉𝒂𝒏 500+ 𝒕𝒓𝒂𝒊𝒏𝒊𝒏𝒈 𝒔𝒆𝒔𝒔𝒊𝒐𝒏𝒔 in the areas of 𝐒𝐨𝐟𝐭𝐰𝐚𝐫𝐞 𝐃𝐞𝐯𝐞𝐥𝐨𝐩𝐦𝐞𝐧𝐭, 𝐃𝐚𝐭𝐚 𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫𝐢𝐧𝐠, 𝐂𝐥𝐨𝐮𝐝, 𝐃𝐚𝐭𝐚 𝐀𝐧𝐚𝐥𝐲𝐬𝐢𝐬, 𝐃𝐚𝐭𝐚 𝐕𝐢𝐬𝐮𝐚𝐥𝐢𝐳𝐚𝐭𝐢𝐨𝐧𝐬, 𝐀𝐫𝐭𝐢𝐟𝐢𝐜𝐢𝐚𝐥 𝐈𝐧𝐭𝐞𝐥𝐥𝐢𝐠𝐞𝐧𝐜𝐞 𝐚𝐧𝐝 𝐌𝐚𝐜𝐡𝐢𝐧𝐞 𝐋𝐞𝐚𝐫𝐧𝐢𝐧𝐠. I am interested in 𝐰𝐫𝐢𝐭𝐢𝐧𝐠 𝐛𝐥𝐨𝐠𝐬, 𝐬𝐡𝐚𝐫𝐢𝐧𝐠 𝐭𝐞𝐜𝐡𝐧𝐢𝐜𝐚𝐥 𝐤𝐧𝐨𝐰𝐥𝐞𝐝𝐠𝐞, 𝐬𝐨𝐥𝐯𝐢𝐧𝐠 𝐭𝐞𝐜𝐡𝐧𝐢𝐜𝐚𝐥 𝐢𝐬𝐬𝐮𝐞𝐬, 𝐫𝐞𝐚𝐝𝐢𝐧𝐠 𝐚𝐧𝐝 𝐥𝐞𝐚𝐫𝐧𝐢𝐧𝐠 new subjects.
Comments
Join the discussionNo comments yet. Be the first to comment.


