Customize Eclipse for Hadoop Programming

from SRC to executable using eclipse

(http://ebiquity.umbc.edu/Tutorials/Hadoop/00%20-%20Intro.html)

1.install the plugin (from hadoop/contrib/ to eclipse/plugin)

2.configure DFS loc:

-NameNode: http://localhost:50070/

-JobTracker: http://localhost:50030/

??can't browsing DFS in eclipse

-Error: null in local_hadoop

-be aware that you need to spawn Eclipse from within Cygwin in order to access HDFS. It seems that the plugin uses "whoami" to get info about the active user.

3.create the helloworld project.

-specify the home loc of hadoop installation.

-create Driver class, and modify the outdated code to,

conf.setInputFormat(TextInputFormat.class);

conf.setOutputFormat(TextOutputFormat.class);

FileInputFormat.setInputPaths(conf, new Path("In"));

FileOutputFormat.setOutputPath(conf, new Path("Out"));

??can't run as (seems a bug of plugin-0.18.3 with Eclipse 3.3+: http://trac.nchc.org.tw/cloud/wiki/waue/Hadoop_Eclipse)

-download Eclipse 3.3(only works with 3.3, still problematic with 3.2...)

4.run a MR program in cmd
(http://hadoop.apache.org/core/docs/current/mapred_tutorial.html#Example%3A+WordCount+v1.0)
$ mkdir wordcount_classes
$ javac -classpath ${HADOOP_HOME}/hadoop-${HADOOP_VERSION}-core.jar -d wordcount_classes WordCount.java
$ jar -cvf /usr/joe/wordcount.jar -C wordcount_classes/ .
-upload files
$ bin/hadoop jar /usr/joe/wordcount.jar org.myorg.WordCount input output

PS:
Why eclipse with hadoop plugin must be spawned from cygwin.
-each process has a pool of environmental variables (PATH), presence of which implies enabling of certain executables (shell commends).
-eclipse process spawning from CYGWIN has same PATH as CYGWIN, so can use WHOISME cmd.

Fireup a Hadoop Cluster

to install Hadoop
-install ssh(as in cygwin installation) in Windows
-download hadoop and specify the JAVA_HOME;!no space in path!;
(http://hadoop.apache.org/core/docs/current/quickstart.html#Download)

to decide which mode to run: 3 options:
1.standalone operations
-run in one Java process, and no need bother to use HDFS.
2.Pseudo-Distributed Mode
-multiple Hadoop daemons, each running in a separate Java process, but all on one node(no network messages).
-need HDFS to load/store data
3.Fully-Distributed Mode (multiple prcesses, multiple nodes)

to use hadoop in standalone mode
$ mkdir input
$ cp conf/*.xml input
$ bin/hadoop jar hadoop-*-examples.jar grep input output 'dfs[a-z.]+'
$ cat output/*

to start hadoop in Pseudo-Distributed Mode
1.to do configuration in conf/hadoop.xml, rather than in conf/core,hdfs,mapred-site.xml(as in web 'quick start'). (otherwise, NPE).

<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>mapred.job.tracker</name>
<value>localhost:9001</value>
</property>
</configuration>

2.start sshd service
(http://ist.uwaterloo.ca/~kscully/CygwinSSHD_W2K3.html)
-note to add (before ssh-host-config)
-$ chmod +r /etc/group
-$ chmod +r /etc/passwd
-$ chmod +rwx /var
-to start by '$ net start sshd'

2.start hadoop
$ bin/hadoop namenode -format
$ bin/start-dfs.sh
$ bin/start-mapred.sh
and we should monitor by
-namenode: http://localhost:50070/
-jobtracker: http://localhost:50030/

3.excution
$ bin/hadoop fs -put conf input (otherwise,org.apache.hadoop.mapred.InvalidInputException: hdfs://localhost:9000/user/v-yuztan/input)
$ bin/hadoop jar hadoop-*-examples.jar grep input output 'dfs[a-z.]+'
-get output

$ bin/hadoop fs -cat output/*

-or

$ bin/hadoop fs -get output output
$ cat output/*

-when you're done,

$ bin/stop-all.sh

PS:
restarting Hadoop
?Incompatible problem.
(http://issues.apache.org/jira/browse/HADOOP-1212)
-delete all datanode files in dfs.data.dir directory. (tentative solution)
$ bin/hadoop namenode format

rerun hadoop program
?FileAlreadyExistsException
(http://www.cs.brandeis.edu/~cs147a/lab/hadoop-troubleshooting/)
-delete the output dir...