HDF5 Plugin

The `HDF5`_ plugin is designed to support both MIF and SSF parallel I/O paradigms. In SSF mode, writes can be collective or independent. In MIF mode, only independent writes are supported because each task creates its own datasets.

In addition, it supports Gzip, szip and zfp compression but only in MIF mode.

Finally, there are a number of command-line arguments to the plugin that allow control of a number of low-level `HDF5`_ features. More can be easily added.

There have been some recent additions to help improve scalable performance of the HDF5_ library and support for these features are yet to have been added to the plugin.

There is a minor difference in the representation of the mesh and its field between the MIF and SSF parallel I/O modes. In MIF mode, each task outputs a chunk of mesh such that nodes on the exterior of the chunk are duplicates of the equivalent nodes on neighboring chunks. However, in SSF mode, the whole mesh is assembled into a single, monolithic whole mesh. The nodes that are duplicated in neighboring mesh chunks are factored out of the actual write requests to HDF5 to avoid lower levels in the I/O stack from having to handle coincident writes.

Command-Line Arguments

Note

Need to auto-gen the command-line args doc from the source code or a run of the executable with –help

Dependencies

To use either gzip or szip compression with the `HDF5`_ plugin, the HDF5 library to which MACSio is linked must have been compiled with support for zlib and szip.

However, to use zfp compression with the `HDF5`_ plugin, all that is necessary is that the H5Z-ZFP filter be available to the `HDF5`_ library to load at run time as any other filter plugin. The H5Z-ZFP filter must have been compatably compiled. In addition, the environment variable, HDF5_PLUGIN_PATH set to include a directory containining the H5Z-ZFP filter plugin library. For example,

env HDF5_PLUGIN_PATH=<path-to-plugin-dir> mpirun -np 4 ./macsio ...

Defines

H5Pset_zfp_rate_cdata(R, N, CD)

H5Z-ZFP generic interface for setting rate mode.

H5Pset_zfp_precision_cdata(P, N, CD)

H5Z-ZFP generic interface for setting precision mode.

H5Pset_zfp_accuracy_cdata(A, N, CD)

H5Z-ZFP generic interface for setting accuracy mode.

H5Pset_zfp_expert_cdata(MiB, MaB, MaP, MiE, N, CD)

H5Z-ZFP generic interface for setting expert mode.

MAINZER_PARAMS

Typedefs

typedef struct _user_data user_data_t

User data for MIF callbacks.

Functions

hid_t make_fapl()

create HDF5 library file access property list

int get_tokval(char const *src_str, char const *token_to_match, void *val_ptr)

Utility to parse compression string command-line args.

Parameters

  • src_str: CL arg string to be parsed

  • token_to_match: a token in the string to be matched including a trailing scanf format specifier

  • val_ptr: Pointer to memory where parsed value should be placed

Does a case-insensitive search of src_str for token_to_match not including the trailing format specifier. Upon finding a match, performs a scanf at the location of the match into temporary memory confirming the scan will actually succeed. Then, performs the scanf again, storying the result to the memory indicated in val_ptr.

Return

0 on error, 1 on success

hid_t make_dcpl(char const *alg_str, char const *params_str, hid_t space_id, hid_t dtype_id)

create HDF5 library dataset creation property list

Parameters

  • alg_str: compression algorithm string

  • params_str: compression params string

  • space_id: HDF5 dataspace id for the dataset

  • dtype_id: HDF5 datatype id for the dataset

If the dataset size is below the minsize threshold, no special storage layout or compression action is taken.

Chunking is initially set to single-chunk. However, for szip compressor, chunking can be set by command-line arguments.

int process_args(int argi, int argc, char *argv[])

Process command-line arguments an set local variables.

Parameters

  • argi: argument index to start processing argv

  • argc: argc from main

  • argv: argv from main

void main_dump_sif(json_object *main_obj, int dumpn, double dumpt)

Single shared file implementation of main dump.

Parameters

  • main_obj: main json data object to dump

  • dumpn: dump number (like a cycle number)

  • dumpt: dump time

void *CreateHDF5File(const char *fname, const char *nsname, void *userData)

MIF create file callback for HDF5 MIF mode.

Parameters

  • fname: file name

  • nsname: curent task namespace name

  • userData: user data specific to current task

void *OpenHDF5File(const char *fname, const char *nsname, MACSIO_MIF_ioFlags_t ioFlags, void *userData)

MIF Open file callback for HFD5 plugin MIF mode.

Parameters

  • fname: filename

  • nsname: namespace name for current task

  • userData: task specific user data for current task

int CloseHDF5File(void *file, void *userData)

MIF close file callback for HDF5 plugin MIF mode.

Parameters

  • file: void* to hid_t of file to cose

  • userData: task specific user data

void write_mesh_part(hid_t h5loc, json_object *part_obj)

Write individual mesh part in MIF mode.

Parameters

  • h5loc: HDF5 group id into which to write

  • part_obj: JSON object for the mesh part to write

void main_dump_mif(json_object *main_obj, int numFiles, int dumpn, double dumpt)

Main dump output for HDF5 plugin MIF mode.

Parameters

  • main_obj: main data object to dump

  • numFiles: MIF file count

  • dumpn: dump number (like a cycle number)

  • dumpt: dump time

void main_dump(int argi, int argc, char **argv, json_object *main_obj, int dumpn, double dumpt)

Main dump callback for HDF5 plugin.

Parameters

  • argi: arg index at which to start processing argv

  • argc: argc from main

  • argv: argv from main

  • main_obj: main json data object to dump

  • dumpn: dump number

  • dumpt: dump time

Selects between MIF and SSF output.

int register_this_interface()

Function called during static initialization to register the plugin.

Variables

char const *iface_name = "hdf5"

name of this plugin

char const *iface_ext = "h5"

file extension for files managed by this plugin

int use_log = 0

Use HDF5’s logging fapl

int no_collective = 0

Use HDF5 independent (e.g. not collective) I/O

int no_single_chunk = 0

disable single chunking

int silo_block_size = 0

block size for silo block-based VFD

int silo_block_count = 0

block count for silo block-based VFD

int sbuf_size = -1

HDF5 library sieve buf size

int mbuf_size = -1

HDF5 library meta blocck size

int rbuf_size = -1

HDF5 library small data block size

int lbuf_size = 0

HDF5 library log flags

const char *filename
hid_t fid
hid_t dspc = -1
int show_errors = 0
char compression_alg_str[64]
char compression_params_str[512]
int dummy = register_this_interface()

Static initializer statement to cause plugin registration at link time.

this one statement is the only statement requiring compilation by a C++ compiler. That is because it involves initialization and non constant expressions (a function call in this case). This function call is guaranteed to occur during initialization (that is before even ‘main’ is called) and so will have the effect of populating the iface_map array merely by virtue of the fact that this code is linked with a main.

struct _user_data

User data for MIF callbacks.

Public Members

hid_t groupId

HDF5 hid_t of current group