Java – ChainReducer.setReducer method throws error “ChainReducer is not applicable for the arguments”

ChainReducer.setReducer method throws error “ChainReducer is not applicable for the arguments”… here is a solution to the problem.

ChainReducer.setReducer method throws error “ChainReducer is not applicable for the arguments”

I have two mapper classes. So use the ChainMapper.addMapper method to add Mapper, and use the ChainReducer.setReduce method to set up Reducer. The ChainMapper.addMapper method works fine, but the Chain.setReducer method throws a syntax error

The method setReducer(Job, Class<? extends Reducer>, Class<?>, Class<?>, Class<?>, Class<?>, Configuration) in the type ChainReducer is not applicable for the arguments (JobConf, Class<FileComparisionReduce>, Class<LongWritable>, Class<Text>, Class<LongWritable>, Class<Text>, boolean, JobConf)

Here is my driver class :

package fileComparision;

import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.JobClient;
import org.apache.hadoop.mapred.JobConf;
import org.apache.hadoop.mapred.FileInputFormat;
import org.apache.hadoop.mapred.FileOutputFormat;
import org.apache.hadoop.mapred.MapReduceBase;
import org.apache.hadoop.mapred.Mapper;
import org.apache.hadoop.mapred.lib.ChainMapper;
import org.apache.hadoop.mapreduce.lib.chain.ChainReducer;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;

public class DriverComparision extends Configured implements Tool{

@Override
    public int run(String[] arg0) throws Exception {
        JobConf conf = new JobConf(true);
        conf.setJobName("Comaprision of 2 file ");
        JobConf Mapper1 = new JobConf(false);

ChainMapper.addMapper(conf, FileComparisionMapper1.class, LongWritable.class, Text.class, LongWritable.class, Text.class, true, Mapper1);

JobConf Mapper2 = new JobConf(false);
        ChainMapper.addMapper(conf, FileComparisionMapper2.class, LongWritable.class, Text.class, LongWritable.class, Text.class, true, Mapper2);

JobConf Reduc = new JobConf(false);
        ChainReducer.setReducer(conf, FileComparisionReduce.class, LongWritable.class, Text.class, LongWritable.class, Text.class, true , Reduc);

FileInputFormat.setInputPaths(conf, new Path(arg0[0]));
        FileOutputFormat.setOutputPath(conf, new Path(arg0[1]));

conf.setMapOutputKeyClass(LongWritable.class);
        conf.setMapOutputValueClass(Text.class);
        conf.setOutputKeyClass(LongWritable.class);
        conf.setOutputValueClass(Text.class);

JobClient.runJob(conf);

return 0;
    }

Also try to remove the boolean parameter “true”

JobConf Reduc = new JobConf(false);
        ChainReducer.setReducer(conf, FileComparisionReduce.class, LongWritable.class, Text.class, LongWritable.class, Text.class, true , Reduc);

Solution

Finally found a solution.
The wrong package was imported, ie. Import org.apache.hadoop.mapreduce.lib.chain.ChainReducer; instead of import org.apache.hadoop.mapred.lib.ChainReducer;

Related Problems and Solutions