HDFS Concept
-
Cooperation
HDFS have several types of nodes to cooperate the handling of big data.
image.png - Resiliency
HDFS stored its data multiple copies in different Data nodes, so you can still retrieve your data when one nodes burnt down. Name Node is often backed up in local disk and NFS for resiliency. There are also secondary Name Node for backup, and Zookeeper tracks active name nodes for management. - Federation
HDFS's Name node manages specific volume for the corresponding data node. -
Use of HDFS
image.png
HDFS Usage
-
First lanching Hortonworks Docker :
image.png
1)Web Interface
-
Then log into Ambari port which is 127.0.0.1:8080, and we can get into HDFS virtual machine web interface:
image.png -
Switching to the File View and we can create folder/ upload data/ view data
image.png
image.png
2)Command Line Interface
Loggin using command:
ssh "your user name"@127.0.0.1 -p 2222Code are similar with Linux shell:
hadoop fs -ls
hadoop fs -mkdir ml-100k
wget http://media.sundog-soft.com/hadoop/ml-100k/u.data
hadoop fs -copyFromLocal u.data ml-100k/u.data
hadoop fs -rm ml-100/u.data
hadoop fs -rmdir ml-100k
You can also find command functions by typing:
hadoop fs
Exit
exit
