HDFS path to Spark commit and Flink on YARN… here is a solution to the problem.
HDFS path to Spark commit and Flink on YARN
I
use Cloudera Live VMs and I have a Hadoop and SPRAL standalone cluster.
Now I want to submit my work with the spark submit and flink run scripts. This also works. But my app can find the paths to the input and output files in hdfs.
The path I set is as follows: hdfs://127.0.0.1:50010/user/cloudera/outputs
Information I get from this port:
How do I set the path to HDFS in Java?
Best regards,
Paul
Solution
You do not have to set the path to the DataNode host. On the Overview page, you’ll see the connection information for NameNode, which is where you must connect to.