Java – Spark stream output is not saved to an HDFS file

Spark stream output is not saved to an HDFS file… here is a solution to the problem.

Spark stream output is not saved to an HDFS file

I’m trying to save Spark stream output to a file on HDFS. Now, it doesn’t save any files.

Here is my code :

StreamingExamples.setStreamingLogLevels();

SparkConf sparkConf = new SparkConf().setAppName("MyTestCOunt");
JavaStreamingContext ssc = new JavaStreamingContext(sparkConf,  new Duration(1000));

JavaReceiverInputDStream<String> lines = ssc.socketTextStream(args[0], Integer.parseInt(args[1]), StorageLevels.MEMORY_AND_DISK_SER);
JavaDStream<String> words = lines.flatMap(new FlatMapFunction<String, String>() {
                @Override
                public Iterable<String> call(String x) {
                 return  Lists.newArrayList(SPACE.split(x));
                }
      });

JavaPairDStream<String, Integer> wordCounts = words.mapToPair(
      new PairFunction<String, String, Integer>() {
        @Override
        public Tuple2<String, Integer> call(String s) {
                 return new Tuple2<String, Integer>(s, 1);
        }
      }).reduceByKey(new Function2<Integer, Integer, Integer>() {
        @Override
        public Integer call(Integer i1, Integer i2) {
          return i1 + i2;
        }
  }); 

wordCounts.print();
wordCounts.saveAsHadoopFiles("hdfs://mynamenode:8020/user/spark/mystream/","abc");
ssc.start();
ssc.awaitTermination();

wordCounts.print() works, but wordCounts.saveAsHadoopFiles doesn’t, any ideas?

I’m running the following command:

1) nc -lk 9999

2) ./bin/run-example org.apache.spark.examples.streaming.NetworkWordCount localhost 9999

Thanks in advance: !!!

Solution

I solved the same problem by specifying master as local[x] x > 1. If you’re running master locally, Spark can’t allocate slots to execute tasks.
Like

SparkConf conf = new SparkConf().setAppName("conveyor").setMaster("local[4]");

Related Problems and Solutions