Java – How to use Java + LibGDX to efficiently read float’s text files?

How to use Java + LibGDX to efficiently read float’s text files?… here is a solution to the problem.

How to use Java + LibGDX to efficiently read float’s text files?

I’m writing a small scientific data visualization application using Java/LibGDX. The application first reads a text file describing a 4-dimensional array:

188 225 3 6 0.001 -0.023 1.345 2.475 ... 4.327

The first four integers specify the dimensions of the array: 188 x 255 x 3 x 6, followed by 188 x 255 x 3 x 6 ~= 760000 numbers. In total, the text file takes up about 5MB of space.

The 4D array represents an “enhanced” image that is 188 pixels high and 255 pixels wide, with 3 color channels (RGB). Each pixel channel has 6 “parameters” instead of one in a regular RGB image.

I read the space-separated list into memory using the following copy from the official docs:

FileHandle ptmFile = Gdx.files.internal(filename);
String ptmText= ptmFile.readString();

Then I created a simple wrapper class AugPixel, which is actually a wrapper for a 6-element array of each pixel. I use a scanner to read and parse each number from a string (excuse my semi-pseudocode):

Scanner scan = new Scanner(ptmText);
int nRows = scan.nextInt();
int nCols = scan.nextInt();
int nColors = scan.nextInt();
int nParams = scan.nextInt();

AugPixel[][][] im = new AugPixel[nRows][nCols][nColors];

for (i = 1~nRows)
  for (j = 1~nCols)
    for (k = 1~nColors)
      im[i][j][k] = new AugPixel(nParams);
      for (m = 1~nParams) 
        im[i][j][k].addParam(scan.nextFloat());

This method works for the desktop version. It takes about 5 seconds to load the file and build all AugPixel objects. However, when I move the app to an actual android device (Nexus 7 32G (2nd generation)), the loading process takes 5 minutes!

So my question is: is there any way to speed up the loading process? I am free to redefine the organization of files as I see fit.

Thanks!

Solution

If reading data from a file is the main bottleneck, you can try some form of asynchronous IO so that you can process partial strings while loading the rest.

If the main bottleneck is parsing strings, you can try multithreading or switch to a binary file format. If you are willing to ditch the text format, the best solution may be to switch to a binary file format, which should drastically reduce the processing time and possible file size.

Related Problems and Solutions