Turning Compression On or Off on Directories

You can turn compression on or off for a given directory in two ways:

  • Set the value of the Compression attribute in the .dfs_attributes file at the top level of the directory.
    • Set Compression=lzf|lz4|zlib to turn compression on for a directory.
    • Set Compression=false to turn compression off for a directory.
  • Use the command hadoop mfs -setcompression on|off/lzf/lz4/zlib <dir>.

If you choose -setcompression on without specifying an algorithm, lz4 is used by default. This algorithm has improved compression speeds for MapR's block size of 64 KB.


Suppose the volume test is NFS-mounted at /mapr/my.cluster.com/projects/test. You can turn off compression by editing the file /mapr/my.cluster.com/projects/test/.dfs_attributes and setting Compression=false. To accomplish the same thing from the hadoop shell, use the following command:

hadoop mfs -setcompression off /projects/test

You can view the compression settings for directories using the hadoop mfs -ls command. For example,

# hadoop mfs -ls /
        Found 23 items
        vrwxr-xr-x Z   - root root         13 2012-04-29 10:24  268435456 /.rw
        p mapr.cluster.root writeable 2049.35.16584 -> 2049.16.2  scale-50.scale.lab:5660 scale-51.scale.lab:5660 scale-52.scale.lab:5660
        vrwxr-xr-x U   - root root          7 2012-04-28 22:16   67108864 /hbase
        p mapr.hbase default 2049.32.16578 -> 2050.16.2  scale-50.scale.lab:5660 scale-51.scale.lab:5660 scale-52.scale.lab:5660
        drwxr-xr-x Z   - root root          0 2012-04-29 09:14  268435456 /tmp
        p 2049.41.16596  scale-50.scale.lab:5660 scale-51.scale.lab:5660 scale-52.scale.lab:5660
        vrwxr-xr-x Z   - root root          1 2012-04-27 22:59  268435456 /user
        p users default 2049.36.16586 -> 2055.16.2  scale-50.scale.lab:5660 scale-52.scale.lab:5660 scale-51.scale.lab:5660
        drwxr-xr-x Z   - root root          1 2012-04-27 22:37  268435456 /var
        p 2049.33.16580  scale-50.scale.lab:5660 scale-51.scale.lab:5660 scale-52.scale.lab:5660 

The symbols for the various compression settings are explained here:


Compression Setting








Uncompressed, or previously compressed by another algorithm