Securely Providing ADLS Credentials

You can provide your ADLS credentials securely by hiding the open, readable configuration on the command line using the Hadoop credential provider.

  1. Generate a jceks file for ADLS authorization:
    hadoop credential create dfs.adls.oauth2.client.id -provider jceks://hdfs/user/USER_NAME/adlskeyfile.jceks -value client ID
    hadoop credential create dfs.adls.oauth2.credential -provider jceks://hdfs/user/USER_NAME/adlskeyfile.jceks -value client secret
    hadoop credential create dfs.adls.oauth2.refresh.url -provider jceks://hdfs/user/USER_NAME/adlskeyfile.jceks -value refresh URL
  2. Run the DistCp example using the jceks file:
    hadoop distcp
    [-D hadoop.security.credential.provider.path=localjceks://hdfs/user/USER_NAME/adlskeyfile.jceks]
    hdfs://<NameNode Hostname>:9001/user/foo/007020615
    adl://<Account Name>.azuredatalakestore.net/testDir/
  3. Configure the core-site.xml file to use the jceks file:
    <property>
      <name>hadoop.security.credential.provider.path</name>
      <value>localjceks://hdfs/user/USER_NAME/adlskeyfile.jceks</value>
      <description>Path to interrogate for protected credentials.</description>
    </property>