Why does the data node shut down when running hadoop?… here is a solution to the problem.
Why does the data node shut down when running hadoop?
I installed hadoop 1.0.4 on VirtualBox’s ubuntu 11.0.4 (same as my hostname) and the data node somehow shuts down, I get the following error in the log file
/************************************************************
STARTUP_MSG: Starting DataNode
STARTUP_MSG: host = VirtualBox/127.0.1.1
STARTUP_MSG: args = []
STARTUP_MSG: version = 1.0.4
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r 1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012
************************************************************/
2013-08-18 19:52:21,301 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2013-08-18 19:52:21,394 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered.
2013-08-18 19:52:21,412 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2013-08-18 19:52:21,417 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics system started
2013-08-18 19:52:23,207 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi registered.
2013-08-18 19:52:23,276 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists!
2013-08-18 19:52:26,887 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Incompatible namespaceIDs in /app/hadoop/tmp/dfs/data: namenode namespaceID = 457871323; datanode namespaceID = 2066655210
at org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:232)
at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:147)
at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:385)
at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:299)
at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1582)
at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)
at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)
at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)
2013-08-18 19:52:26,903 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at VirtualBox/127.0.1.1
************************************************************/
You know why? How do I fix it?
Solution
I’ve encountered this situation a few times. If restarting the data node does not help, do the following:
- Restart Hadoop
- Go to /app/hadoop/tmp/dfs/name/current
- Open the version (i.e. via
vim VERSION
). - Record the namespaceID
- Go to /app/hadoop/tmp/dfs/data/current
- Open the version (i.e. via
vim VERSION
). - Replace the namespaceID with the namespaceID that you recorded in step 4.
This should fix the problem.