Java – HDFS path to Spark commit and Flink on YARN

HDFS path to Spark commit and Flink on YARN… here is a solution to the problem.

HDFS path to Spark commit and Flink on YARN

I

use Cloudera Live VMs and I have a Hadoop and SPRAL standalone cluster.
Now I want to submit my work with the spark submit and flink run scripts. This also works. But my app can find the paths to the input and output files in hdfs.
The path I set is as follows: hdfs://127.0.0.1:50010/user/cloudera/outputs
Information I get from this port:
enter image description here

How do I set the path to HDFS in Java?

Best regards,
Paul

Solution

You do not have to set the path to the DataNode host. On the Overview page, you’ll see the connection information for NameNode, which is where you must connect to.

Related Problems and Solutions