Wednesday, August 14, 2013

Don't call filesystem.close in Hadoop

Recently the MapReduce jobs in my project suddenly failed. It turned out that my colleague added a line in the code which closes the filesystem.

val path = new Path("/user/bewang/data")
val fs = path.getFileSystem(conf)
fs.mkdirs(path)
fs.close
Nothing seems wrong. We were told to clean up the mess you created. After using it, just close it. Is it a good practice? Unfortunately it doesn't work in Hadoop world. Actually Hadoop client manage all connections to Hadoop cluster. If you call fs.close(), the connections to the cluster are broken, and you cannot do anything after that. Don't call close, let Hadoop handle it.

3 comments:

  1. tutorials on Upgrade Hadoop is excellent.I am happy to found such helpful and fascinating post that is written in well manner. i actually enhanced my data when browse your post .thanks.
    Hadoop Training in hyderabad

    ReplyDelete
  2. This comment has been removed by the author.

    ReplyDelete
  3. Hi admin thanks for sharing informative article on hadoop technology. In coming years, hadoop and big data handling is going to be future of computing world. This field offer huge career prospects for talented professionals. Thus, taking Hadoop & Spark Training in Hyderabad will help you to enter big data hadoop & spark technology.

    ReplyDelete