Conflicting APIs when trying to run MRUnit samples
I’ve been working on MRUnit and trying to follow the > tutorial runs it for the Hadoop WordCount sample unit testing
I’m not a fan though, I’ve been running code with Eclipse, but I’m getting errors with the setMapper function all the time
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mrunit.mapreduce.MapDriver;
import org.apache.hadoop.mrunit.mapreduce.MapReduceDriver;
import org.apache.hadoop.mrunit.mapreduce.ReduceDriver;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.junit.Before;
import org.junit.Test;
public class TestWordCount {
MapReduceDriver<LongWritable, Text, Text, IntWritable, Text, IntWritable> mapReduceDriver;
MapDriver<LongWritable, Text, Text, IntWritable> mapDriver;
ReduceDriver<Text, IntWritable, Text, IntWritable> reduceDriver;
@Before
public void setUp() throws IOException
{
WordCountMapper mapper = new WordCountMapper();
mapDriver = new MapDriver<LongWritable, Text, Text, IntWritable>();
mapDriver.setMapper(mapper); <--Issue here
WordCountReducer reducer = new WordCountReducer();
reduceDriver = new ReduceDriver<Text, IntWritable, Text, IntWritable>();
reduceDriver.setReducer(reducer);
mapReduceDriver = new MapReduceDriver<LongWritable, Text, Text, IntWritable, Text, IntWritable>();
mapReduceDriver.setMapper(mapper); <--Issue here
mapReduceDriver.setReducer(reducer);
}
Error message:
java.lang.Error: Unresolved compilation problems:
The method setMapper(Mapper<LongWritable,Text,Text,IntWritable>) in the type MapDriver<LongWritable,Text,Text,IntWritable> is not applicable for the arguments ( WordCountMapper)
The method setMapper(Mapper<LongWritable,Text,Text,IntWritable>) in the type MapReduceDriver<LongWritable,Text,Text,IntWritable,Text,IntWritable> is not applicable for the arguments (WordCountMapper)
Looking up this issue, I thought it might be an API conflict, but I’m not sure where to look for it. Has anyone else encountered this issue before?
Edit I’m using a user-defined library that contains the hadoop2 jar and the latest Junit(4.10) jar.
Edit 2 This is the code for WordCountMapper
import java.io.IOException;
import java.util.StringTokenizer;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;
public class WordCountMapper extends Mapper<Object, Text, Text, IntWritable>
{
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(Object key, Text value, Context context)throws IOException, InterruptedException
{
StringTokenizer itr = new StringTokenizer(value.toString());
while (itr.hasMoreTokens())
{
word.set(itr.nextToken());
context.write(word, one);
}
}
}
Final edited/valid
Turns out I need to set it up
WordCountMapper mapper = new WordCountMapper();
to
Mapper mapper = new WordCountMapper();
Because there is a problem with generics. I also need to import the mockito library into my user-defined library.
Solution
This is your problem
public class WordCountMapper extends Mapper<Object, Text, Text, IntWritable>
....
MapDriver<LongWritable, Text, Text, IntWritable> mapDriver;
Your WordCountMapper
input type (Object
) is not compatible with the MapDriver
input type (LongWritable
). Change your Mapper
definition to
class WordCountMapper extends Mapper<LongWritable, Text, Text, IntWritable>
You may also want to change the map
method parameter from Object
key to LongWritable key
.