Outils pour utilisateurs

Outils du site


General performance of HDF5

In the case of you don't know yet hdf5, let see a naive comparison between a raw text storage and a hdf5 storage. The test has been done with a grid in 4 dimensions stored as 15000000 bars. Several compression methods have been tested for the hdf5 storage: without compression, LZF and Gzip with compression level to 9 (hdf5 doc).

storage file size writing time reading time
raw text 391Mo NA 113s
hdf5 602Mo 8.5s 1.7s
hdf5/lzf 15Mo 8.3s 1.2s
hdf5/gzip 4.3Mo 18.8s 1.5s

Cost for storing viable points

Is it better to store all the points of a grid with its label true or false (viable or not) or to store only the coordinates of the viable points?

I've tried with the lake model sample (under samples/lake on the repository). For the parameters used, 62,8% of the points of the grid was viable, and we get the following results:

  • list of bars (min and max for each coordinates on dimension-reduced space) : 1.8ko
  • grid of true/false: 14ko
  • list of viable points coordinates: 273ko
hdf5experiments.txt · Dernière modification: 2015/11/20 15:57 par nicolas.dumoulin@irstea.fr