In the case of you don't know yet hdf5, let see a naive comparison between a raw text storage and a hdf5 storage. The test has been done with a grid in 4 dimensions stored as 15000000 bars. Several compression methods have been tested for the hdf5 storage: without compression, LZF and Gzip with compression level to 9 (hdf5 doc).
|storage||file size||writing time||reading time|
Is it better to store all the points of a grid with its label true or false (viable or not) or to store only the coordinates of the viable points?
I've tried with the lake model sample (under samples/lake on the repository). For the parameters used, 62,8% of the points of the grid was viable, and we get the following results: