Use sqoop to import mysql into hadoop on Mac

reference guide.

1. prerequisites:
1.1 Hadoop, Sqoop version.
$ hadoop version

Hadoop 2.5.2

Sqoop:  sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz

download mysql-connector-java-5.1.36.jar and put in sqoop/lib folder :  http://mvnrepository.com/artifact/mysql/mysql-connector-java/5.1.36

1.2 edit hadoop/etc/hadoop/yarn.site.xml

<configuration>

  <property>

       <name>yarn.nodemanager.aux-services</name>

       <value>mapreduce_shuffle</value>

    </property>

    <property>

       <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>

       <value>org.apache.hadoop.mapred.ShuffleHandler</value>

    </property>

    <property>

       <name>yarn.application.classpath</name>

       <value>

           %HADOOP_HOME%/etc/hadoop,

           %HADOOP_HOME%/share/hadoop/common/*,

           %HADOOP_HOME%/share/hadoop/common/lib/*,

           %HADOOP_HOME%/share/hadoop/hdfs/*,

           %HADOOP_HOME%/share/hadoop/hdfs/lib/*,

           %HADOOP_HOME%/share/hadoop/mapreduce/*,

           %HADOOP_HOME%/share/hadoop/mapreduce/lib/*,

           %HADOOP_HOME%/share/hadoop/yarn/*,

           %HADOOP_HOME%/share/hadoop/yarn/lib/*

       </value>

    </property>

</configuration>

1.3 link missed executable binary:

sudo ln -s /usr/bin/java /bin/java

sudo ln -s /usr/local/bin/mysqldump /usr/bin/mysqldump

1.4 run import command under Sqoop folder:

./bin/sqoop import --connect jdbc:mysql://127.0.0.1:3306/sample --username root --table employers --direct -m 1 --target-dir /user/YOUR_NAME/employers

1.4.1 or use import config file:

./bin/sqoop --options-file /usr/local/sqoop/import.txt --table employers -m 1 --target-dir /user/YOUR_NAME/sample/employers --check-column id --incremental append 

import.txt:

----------------

import
--connect
jdbc:mysql://127.0.0.1/db
--username
root

1.4.2   --fields-terminated-by '|'
check "Output line formatting arguments" for more format usage

1.5 incremental import based on column value or modify date.

./bin/sqoop import --connect jdbc:mysql://127.0.0.1:3306/sample --username root --table employers --direct -m 1 --target-dir /user/YOUR_NAME/employers --check-column id --incremental append --last-value 118996

Setup Spark with Hadoop on Mac

1. Go to download page: http://spark.apache.org/downloads.html and selected spark-1.3.1-bin-hadoop2.6.tgz
2. After download, clicked binary to unzip.
3. link binary to specific path. >ln -s DOWNLOAD/spark-1.3.1-bin-hadoop2.6 /usr/local/spark
4. start Hadoop. > start-dfs.sh
5. start Spark. > ./bin/spark-shell
6. visit Spark web ui. http://localhost:4040/jobs/

setup hadoop single node cluster on mac

reference: Apache Hadoop - Setting up a Single Node Cluster

After setup in reference above is done, better to update hadoop config to make it works well after reboot.
1. specified space for namenode and datanode, or they will be put in /tmp, which will be lost after reboot.

$ vi etc/hadoop/hdfs-site.xml

added following property:

    <property>
       <name>dfs.namenode.name.dir</name>
       <value>~/hadoop/namenode</value>
    </property>

    <property>
        <name>dfs.datanode.data.dir</name>
        <value>~/hadoop/data</value>
    </property>

2. disabled all permission check.

$ vi etc/hadoop/hdfs-site.xml
<property>
<name>dfs.permissions</name> 
<value>false</value>
</property>

3. format namenode

$ hdfs namenode -format

4. start hadoop and yarn

$ start-dfs.sh && start-yarn.sh

5. visit web ui
http://localhost:50070/
http://localhost:8088/