MapR 5.0 Documentation : Accessing MapR-FS in Java Applications

This page describes how to access MapR-FS in a Java program, and includes sample code. This page contains the following topics:

As a high-performance filesystem, portions of the MapR-FS file client are based on a native maprfs library. When developing an application, specifying dependence on the JAR file that includes the maprfs library allows you to build applications without having to manage platform-specific dependencies.

Writing a Java Application

In your Java application, you will use a Configuration object to interface with MapR-FS. When you instantiate a Configuration object, it is created with values from Hadoop configuration files.

When the program is build with JAR files from the MapR installation, the Hadoop 1 configuration files are in $MAPR_HOME/hadoop/hadoop-<version>/conf and the Hadoop 2 configuration files are in $HADOOP_HOME/etc/hadoop. This Hadoop configuration directory is in the hadoop classpath that you include when you compile and run the java program.

When the program is built through maven using MapR's maven artifacts, the default Hadoop configuration files are included in the maven artifacts.The user needs to programmatically update the Hadoop configuration to match the Hadoop configuration files on the MapR cluster.

Sample Code

The following sample code shows how to interface with MapR-FS using Java. The example creates a directory, writes a file, then reads the contents of the file.

/* Copyright (c) 2009 & onwards. MapR Tech, Inc., All rights reserved */

//package com.mapr.fs;

import java.net.*;
import org.apache.hadoop.fs.*;
import org.apache.hadoop.conf.*;

/**
* Assumes mapr installed in /opt/mapr
*
* Compilation:
* javac -cp $(hadoop classpath) MapRTest.java
*
* Run:
* java -cp .:$(hadoop classpath) MapRTest /test
*/
public class MapRTest
{
 public static void main(String args[]) throws Exception {
        byte buf[] = new byte[ 65*1024];
        int ac = 0;
        if (args.length != 1) {
            System.out.println("usage: MapRTest pathname");
        return;
        }

        // maprfs:/// -> uses the first entry in /opt/mapr/conf/mapr-clusters.conf
        // maprfs:///mapr/my.cluster.com/
        // /mapr/my.cluster.com/

        // String uri = "maprfs:///";
        String dirname = args[ac++];

        Configuration conf = new Configuration();
        
        //FileSystem fs = FileSystem.get(URI.create(uri), conf); // if wanting to use a different cluster
        FileSystem fs = FileSystem.get(conf);
        
        Path dirpath = new Path( dirname + "/dir");
        Path wfilepath = new Path( dirname + "/file.w");
        //Path rfilepath = new Path( dirname + "/file.r");
        Path rfilepath = wfilepath;


        // try mkdir
        boolean res = fs.mkdirs( dirpath);
        if (!res) {
                System.out.println("mkdir failed, path: " + dirpath);
        return;
        }

        System.out.println( "mkdir( " + dirpath + ") went ok, now writing file");

        // create wfile
        FSDataOutputStream ostr = fs.create( wfilepath,
                true, // overwrite
                512, // buffersize
                (short) 1, // replication
                (long)(64*1024*1024) // chunksize
                );
        ostr.write(buf);
        ostr.close();

        System.out.println( "write( " + wfilepath + ") went ok");

        // read rfile
        System.out.println( "reading file: " + rfilepath);
        FSDataInputStream istr = fs.open( rfilepath);
        int bb = istr.readInt();
        istr.close();
        System.out.println( "Read ok");
        }
}

Compiling and Running a Java Application

You can compile and run the java application using JAR files from the MapR Maven repository or from the MapR installation.

Using JARs from the MapR Maven Repository

MapR publishes Maven artifacts from version 2.1.2 onward at http://repository.mapr.com/maven/.

When compiling for MapR version 4.0.1, add the following dependency to the project's pom.xml file:

<dependency>
   <groupId>org.apache.hadoop</groupId>
  <artifactId>hadoop-core</artifactId>
  <version>2.4.1-mapr-1408</version>
</dependency>

When compiling for MapR version 4.0.2, add the following dependency to the project's pom.xml file:

<dependency>
   <groupId>org.apache.hadoop</groupId>
  <artifactId>hadoop-core</artifactId>
  <version>1.0.3-mapr-4.0.2</version>
</dependency>

This dependency will pull the rest of the dependencies from MapR's Maven repository the next time you do a mvn clean install.The JAR that includes the maprfs library is a dependency for the hadoop-core artifact.

For a complete list of MapR-provided artifacts and further details, see Maven Repository and Artifacts for MapR.

Using JARs from MapR Installation

The maprfs library is in included in the hadoop classpath. Add the hadoop classpath to the JAVA classpath when you compile and run the java application.

  • Use the following command to compile the sample code:
javac -cp $(hadoop classpath) MapRTest.java
  • Use the following command to run the sample code:

java -cp .:$(hadoop classpath) MapRTest /test