NoClassDefFoundError error in Hadoop… here is a solution to the problem.
NoClassDefFoundError error in Hadoop
So after going through a lot of posts about SO and modifying some java basics, I still get this error
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/io/Writable
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2451)
at java.lang.Class.getMethod0(Class.java:2694)
at java.lang.Class.getMethod(Class.java:1622)
at sun.launcher.LauncherHelper.getMainMethod(LauncherHelper.java:494)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:486)
I know the difference between java.lang.NoClassDefFoundError
and ClassNotFoundException
in Java and understand what usually causes it. In short, because some classes are not available to the program at run time, but are available at compile time. So I don’t get compile-time errors.
I added two classpaths, one to commons-logging-1.1.3.jar
and one to hadoop-core.*jar.
I am very confident that the classpath is correct.
This is the import in my program
import java.io.*;
import org.apache.hadoop.io.SequenceFile;
import org.apache.hadoop.io.SequenceFile.*;
import org.apache.hadoop.io.SequenceFile.Writer;
import org.apache.hadoop.io.*;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.io.Writable;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.*;
Solution
You need to create a JAR from Java code like given here
$ mkdir my_classes
$ javac -classpath $HADOOP_HOME/hadoop-$HADOOP_VERSION-core.jar -d my_classes <name of the main class>
$ jar -cvf <name of the jar> -C my_classes .
Run jar: this way
$HADOOP_HOME/bin/hadoop jar <name of the jar> <name of the main class> <arguments to the program>
See documentation page Hadoop’s jar command description.