Got the chance to work on Spark2.0 data frames on Atom. We have faced the following issues and then lot of investigation, have fixed the issue with the following URL.
http://stackoverflow.com/questions/34196302/the-root-scratch-dir-tmp-hive-on-hdfs-should-be-writable-current-permissions
http://stackoverflow.com/questions/34196302/the-root-scratch-dir-tmp-hive-on-hdfs-should-be-writable-current-permissions
