hadoop DistributedFileSystem thread-safe?… here is a solution to the problem.
hadoop DistributedFileSystem thread-safe?
I’m using Hadoop to write to the data I grab.
I have a spring service called from multiple threads to write something to HDFS.
@Service
public class WriteService
{
public void write(String path, String content)
{
FileSystem fs = FileSystem.get(conf);
}
}
I’m not sure if a FileSystem object can be a member of WriteService, and I haven’t found out if it’s thread-safe.
I’m using the DistributedFileSystem object.
Do you know if it’s thread-safe? Can I use it as a member of my service?
Thanks
Solution
Hadoop DFS uses the so-called WORM-Model.
However, to answer this question, it is generally not safe. You still need to consider concurrency control requirements.