Generally, high-performance storage frameworks, such as RocketMQ and Kafka, are implemented through Java File MMAP when storing logs. So what is Java File MMAP?

What is Java File MMAP

Although since JDK version 1.4, JAVA Memory Mapped Files have been in Java NIO package, but it is still a fairly new concept for many program developers. After the introduction of NIO, Java IO has been quite fast, and the memory mapped file provides the fastest IO operation that Java can achieve. This is why those high-performance Java applications should use Memory Mapped Files to persist data. As an important function of NIO, MMAP method provides us with the ability to map part or all of the file to the memory address space. When this memory area is written with data, it will become a dirty page. The operating system will write these data into the file with a certain algorithm, and our Java program does not need to care about them. This is a key advantage of Memory Mapped Files. Even if your program hangs just after writing to memory, the operating system will still write the data in memory to the file system. Another more prominent advantage is shared memory. Memory mapping files can be accessed by multiple processes at the same time, which plays the role of low latency shared memory.

Performance comparison between Java File MMAP and direct operation file

package com.github.hashZhang.scanfold.jdk.file;
import java.io.RandomAccessFile;
import java.nio.ByteBuffer;
import java.nio.MappedByteBuffer;
import java.nio.channels.FileChannel;
import java.util.Random;

public class FileMmapTest {
    public static void main(String[] args) throws Exception {
        //Record start time
        long start = System.currentTimeMillis();
        //Obtain the Channel of the file through RandomAccessFile. This method is commonly used for files that are read and written randomly. We generally use files that are read and written randomly
        RandomAccessFile randomAccessFile = new RandomAccessFile("./FileMmapTest.txt", "rw");
        FileChannel channel = randomAccessFile.getChannel();
        System.out.println("FileChannel Initialization time:" + (System.currentTimeMillis() - start) + "ms");

        //Memory mapping file. The mode is READ_WRITE, if the file does not exist, it will be created
        MappedByteBuffer mappedByteBuffer1 = channel.map(FileChannel.MapMode.READ_WRITE, 0, 128 * 1024 * 1024);
        MappedByteBuffer mappedByteBuffer2 = channel.map(FileChannel.MapMode.READ_WRITE, 0, 128 * 1024 * 1024);

        System.out.println("MMAPFile Initialization time:" + (System.currentTimeMillis() - start) + "ms");

        start = System.currentTimeMillis();
        testFileChannelSequentialRW(channel);
        System.out.println("FileChannel Sequential read / write time:" + (System.currentTimeMillis() - start) + "ms");

        start = System.currentTimeMillis();
        testFileMMapSequentialRW(mappedByteBuffer1, mappedByteBuffer2);
        System.out.println("MMAPFile Sequential read / write time:" + (System.currentTimeMillis() - start) + "ms");

        start = System.currentTimeMillis();
        try {
            testFileChannelRandomRW(channel);
            System.out.println("FileChannel Random read / write time:" + (System.currentTimeMillis() - start) + "ms");
        } finally {
            randomAccessFile.close();
        }

        //File closing does not affect MMAP writing and reading
        start = System.currentTimeMillis();
        testFileMMapRandomRW(mappedByteBuffer1, mappedByteBuffer2);
        System.out.println("MMAPFile Random read / write time:" + (System.currentTimeMillis() - start) + "ms");
    }


    public static void testFileChannelSequentialRW(FileChannel fileChannel) throws Exception {
            byte[] bytes = "Test string 1 test string 1 test string 1 test string 1 test string 1 test string 1 test string 1 test string 1 test string 1 test string 1 test string 1 test string 1 test string 1".getBytes();
            byte[] to = new byte[bytes.length];
            //Allocate direct memory to reduce replication
            ByteBuffer byteBuffer = ByteBuffer.allocateDirect(bytes.length);
            //Sequential write
            for (int i = 0; i < 100000; i++) {
                byteBuffer.put(bytes);
                byteBuffer.flip();
                fileChannel.write(byteBuffer);
                byteBuffer.flip();
            }

            fileChannel.position(0);
            //Sequential reading
            for (int i = 0; i < 100000; i++) {
                fileChannel.read(byteBuffer);
                byteBuffer.flip();
                byteBuffer.get(to);
                byteBuffer.flip();
            }
    }

    public static void testFileMMapSequentialRW(MappedByteBuffer mappedByteBuffer1, MappedByteBuffer mappedByteBuffer2) throws Exception {
        byte[] bytes = "Test string 2 test string 2 test string 2 test string 2 test string 2 test string 2 test string 2 test string 2 test string 2 test string 2 test string 2 test string 2 test string 2".getBytes();
        byte[] to = new byte[bytes.length];

        //Sequential write
        for (int i = 0; i < 100000; i++) {
            mappedByteBuffer1.put(bytes);
        }
        //Sequential reading
        for (int i = 0; i < 100000; i++) {
            mappedByteBuffer2.get(to);
        }
    }

    public static void testFileChannelRandomRW(FileChannel fileChannel) throws Exception {
        try {
            byte[] bytes = "Test string 1 test string 1 test string 1 test string 1 test string 1 test string 1 test string 1 test string 1 test string 1 test string 1 test string 1 test string 1 test string 1".getBytes();
            byte[] to = new byte[bytes.length];
            //Allocate direct memory to reduce replication
            ByteBuffer byteBuffer = ByteBuffer.allocateDirect(bytes.length);
            //Random write
            for (int i = 0; i < 100000; i++) {
                byteBuffer.put(bytes);
                byteBuffer.flip();
                fileChannel.position(new Random(i).nextInt(bytes.length*100000));
                fileChannel.write(byteBuffer);
                byteBuffer.flip();
            }
            //Random read
            for (int i = 0; i < 100000; i++) {
                fileChannel.position(new Random(i).nextInt(bytes.length*100000));
                fileChannel.read(byteBuffer);
                byteBuffer.flip();
                byteBuffer.get(to);
                byteBuffer.flip();
            }
        } finally {
            fileChannel.close();
        }
    }

    public static void testFileMMapRandomRW(MappedByteBuffer mappedByteBuffer1, MappedByteBuffer mappedByteBuffer2) throws Exception {
        byte[] bytes = "Test string 2 test string 2 test string 2 test string 2 test string 2 test string 2 test string 2 test string 2 test string 2 test string 2 test string 2 test string 2 test string 2".getBytes();
        byte[] to = new byte[bytes.length];

        //Random write
        for (int i = 0; i < 100000; i++) {
            mappedByteBuffer1.position(new Random(i).nextInt(bytes.length*100000));
            mappedByteBuffer1.put(bytes);
        }
        //Random read
        for (int i = 0; i < 100000; i++) {
            mappedByteBuffer2.position(new Random(i).nextInt(bytes.length*100000));
            mappedByteBuffer2.get(to);
        }
    }
}

Here, we initialize a file and map it to 128M of memory. In the way of FileChannel and MMAP, some contents are written and some contents are read through sequential or random reading and writing.

The operation result is:

FileChannel Initialization time: 7 ms
MMAPFile Initialization time: 8 ms
FileChannel Sequential read / write time: 420 ms
MMAPFile Sequential reading and writing time: 20 ms
FileChannel Random reading and writing time: 860 ms
MMAPFile Random reading and writing time: 45 ms

It can be seen that the operation of files through MMAP memory mapped files is faster, and the performance improvement is quite obvious.

WeChat search "my programming meow" attention to the official account, daily brush, easy to upgrade technology, and capture all kinds of offer: