Thursday, July 16, 2015

setup hadoop single node cluster on mac

referenceApache Hadoop - Setting up a Single Node Cluster

After setup in reference above is done, better to update hadoop config to make it works well after reboot.
1. specified space for namenode and datanode, or they will be put in /tmp, which will be lost after reboot.
$ vi etc/hadoop/hdfs-site.xml
added following property:
    <property>
       <name>dfs.namenode.name.dir</name>
       <value>~/hadoop/namenode</value>
    </property>

    <property>
        <name>dfs.datanode.data.dir</name>
        <value>~/hadoop/data</value>
    </property>
2. disabled all permission check.
$ vi etc/hadoop/hdfs-site.xml
<property>
<name>dfs.permissions</name> 
<value>false</value>
</property>
3. format namenode
$ hdfs namenode -format
4. start hadoop and yarn
$ start-dfs.sh && start-yarn.sh
5. visit web ui
http://localhost:50070/
http://localhost:8088/

No comments:

Post a Comment