Fireup a Hadoop Cluster

to install Hadoop
-install ssh(as in cygwin installation) in Windows
-download hadoop and specify the JAVA_HOME;!no space in path!;
(http://hadoop.apache.org/core/docs/current/quickstart.html#Download)

to decide which mode to run: 3 options:
1.standalone operations
-run in one Java process, and no need bother to use HDFS.
2.Pseudo-Distributed Mode
-multiple Hadoop daemons, each running in a separate Java process, but all on one node(no network messages).
-need HDFS to load/store data
3.Fully-Distributed Mode (multiple prcesses, multiple nodes)

to use hadoop in standalone mode
$ mkdir input
$ cp conf/*.xml input
$ bin/hadoop jar hadoop-*-examples.jar grep input output 'dfs[a-z.]+'
$ cat output/*

to start hadoop in Pseudo-Distributed Mode
1.to do configuration in conf/hadoop.xml, rather than in conf/core,hdfs,mapred-site.xml(as in web 'quick start'). (otherwise, NPE).

<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>mapred.job.tracker</name>
<value>localhost:9001</value>
</property>
</configuration>

2.start sshd service
(http://ist.uwaterloo.ca/~kscully/CygwinSSHD_W2K3.html)
-note to add (before ssh-host-config)
-$ chmod +r /etc/group
-$ chmod +r /etc/passwd
-$ chmod +rwx /var
-to start by '$ net start sshd'

2.start hadoop
$ bin/hadoop namenode -format
$ bin/start-dfs.sh
$ bin/start-mapred.sh
and we should monitor by
-namenode: http://localhost:50070/
-jobtracker: http://localhost:50030/

3.excution
$ bin/hadoop fs -put conf input (otherwise,org.apache.hadoop.mapred.InvalidInputException: hdfs://localhost:9000/user/v-yuztan/input)
$ bin/hadoop jar hadoop-*-examples.jar grep input output 'dfs[a-z.]+'
-get output

$ bin/hadoop fs -cat output/*

-or

$ bin/hadoop fs -get output output
$ cat output/*

-when you're done,

$ bin/stop-all.sh

PS:
restarting Hadoop
?Incompatible problem.
(http://issues.apache.org/jira/browse/HADOOP-1212)
-delete all datanode files in dfs.data.dir directory. (tentative solution)
$ bin/hadoop namenode format

rerun hadoop program
?FileAlreadyExistsException
(http://www.cs.brandeis.edu/~cs147a/lab/hadoop-troubleshooting/)
-delete the output dir...