Java Performance – How do I write a large array to disk/SD card with high performance?

Is there a way in Java to write a large array of integers to disk? I’m doing this on Android and haven’t found anything close to native C code yet.

The resulting file does not need to be ported to a different machine in a different representation, so logically it is enough to write a large number of the underlying bytes. But I don’t know how to do this efficiently from Java.

I tried searching online and tested the following:

• Serialization – Very slow as expected.
• With NIO – still slow – Android traces show one operation at a time per integer:

Thanks in advance

NIO code:

int[] array = new array[10000000];

...

raf = new RandomAccessFile(ti.testFileName, "rw");
chan = raf.getChannel();
MappedByteBuffer out = chan.map(FileChannel.MapMode.READ_WRITE, 0, array.length*4);
ib = out.asIntBuffer();
ib.put(array);
out.force();
raf.close();

You say it’s slow, but the speed will most likely depend on the speed of your disk subsystem. You should be able to write 40 MB to a normal disk in about half a second to commit to disk.

The following uses NIO, which takes 665 ms to write and 62 ms on a workstation. Reads and writes shuffle the same amount of data, but reads can fetch data from the operating system cache, with the difference being the time it takes to write to disk.

int[] ints = new int[10 * 1000 * 1000];
long start = System.nanoTime();

ByteBuffer byteBuffer = ByteBuffer.allocateDirect(ints.length*4+4);
byteBuffer.putInt(ints.length);
IntBuffer intBuffer = byteBuffer.asIntBuffer();
intBuffer.put(ints);
byteBuffer.position(0);

FileChannel fc = new FileOutputStream("main.dat").getChannel();
fc.write(byteBuffer);
fc.force(false);
fc.close();
long time = System.nanoTime() - start;
System.out.println("Write time " + time / 1000 / 1000 + " ms.");

long start2 = System.nanoTime();
FileChannel fc2 = new FileInputStream("main.dat").getChannel();
ByteBuffer lengthBuffer = ByteBuffer.allocate(4);
while(lengthBuffer.remaining()>0) fc2.read(lengthBuffer);
int length = lengthBuffer.getInt(0);

int[] ints2 = new int[length];
ByteBuffer buffer2 = ByteBuffer.allocateDirect(length*4);
while(buffer2.remaining()>0 && fc2.read(buffer2) > 0);
buffer2.flip();
buffer2.asIntBuffer().get(ints2);
long time2 = System.nanoTime() - start2;
System.out.println("Read time " + time2 / 1000 / 1000 + " ms.");

I’ve added the length to the beginning of the file, so I don’t have to assume it. By the way: I’ve fixed the error in the write.