Example (macsio_miftmpl.c)

Writing a new plugin for MACSio involves a mininum of two new files in the plugins directory. One is the C or C++ source code for the plugin. The other is a .make file that includes various plugin-specific variable definitions and logic to decode library dependencies that effect.

The miftmpl plugin is intended to serve as a template for how to create a basic MIF mode plugin for MACSio. This template code does indeed actually function correctly as a MACSio plugin. It does so by writing MACSio’s internal JSON objects repesenting each mesh part as ascii strings to the individual files.

Each processor in a MIF group serializes each JSON object representing a mesh part to an ascii string. Then, each of these strings is appended to the end of the file. For each such string, the plugin maintains knowledge of the mesh part’s ID, the filename it was written to and the offset within the file.

The filenames, offsets and mesh part IDs are then written out as a separate JSON object to a root or master file. Currently, there is no plugin in VisIt to read these files and display them. But, this example code does help to outline the basic work to write a MIF plugin.

In practice, this plugin could have simply written the entire JSON object from each processor to its MIF group’s file. However, in doing that, the resulting file would not “know” such things as how many mesh parts there are or where a given mesh part is located in the file set. So, we wind up writing JSON objects for each part individually so that we can keep track of where they all are in the fileset.

Some of the aspects of this plugin code exist here only to serve as an example in writing a MIF plugin and are non-essential to the proper operation of this plugin.

MACSio uses a static load approach to its plugins. The MACSio main executable must be linked with all the plugins it expects to use.

In writing any MACSio plugin (MIF or SSF) be sure to declare all of your plugin’s symbols (functions, local variables, etc.) as static. Each plugin is being linked into MACSio’s main and any symbols that are not static file scope will wind up appearing in and therefore being vulnerable too global namespace collisions. The plugin’s main interface methods to MACSio are handled via registration of a set of function pointers.

Functions

int process_args(int argi, int argc, char *argv[])

Process command-line arguments specific to this plugin.

Parameters
  • argi: Argument index of first argument that is specific to this plugin

  • argc: argc as passed into main

  • argv: argv as passed into main

Uses MACSIO_CLARGS_ProcessCmdline() to do its work.

This example plugin is implemented to route command line arguments to memory locations (e.g. static variables) here in the plugin. Alternatively, a plugin can choose to route the results of MACSIO_CLARGS_ProcessCmdline() to a JSON object. MACSio’s main is implemented that way.

void *CreateMyFile(const char *fname, const char *nsname, void *userData)

CreateFile MIF Callback.

Parameters
  • fname: Name of the MIF file to create

  • nsname: Name of the namespace within the file for caller should use.

  • userData: Optional plugin-specific user-defined data

This implments the MACSIO_MIF_CreateFile callback needed for a MIF mode plugin.

Return

A void pointer to the plugin-specific file handle

void *OpenMyFile(const char *fname, const char *nsname, MACSIO_MIF_ioFlags_t ioFlags, void *userData)

OpenFile MIF Callback.

Parameters
  • fname: Name of the MIF file to open

  • nsname: Name of the namespace within the file caller should use

  • ioFlags: Various flags indicating behavior/options

  • userData: Optional plugin-specific user-defined data

This implments the MACSIO_MIF_OpenFile callback needed for a MIF mode plugin.

Return

A void pointer to the plugin-specific file handle

int CloseMyFile(void *file, void *userData)

CloseFile MIF Callback.

Parameters
  • file: A void pointer to the plugin specific file handle

  • userData: Optional plugin specific user-defined data

This implments the MACSIO_CloseFile callback needed for a MIF mode plugin.

json_object *write_mesh_part(FILE *myFile, char const *fileName, json_object *part_obj)

Write a single mesh part to a MIF file.

Parameters
  • myFile: The file handle being used in a MIF dump

  • fileName: Name of the MIF file

  • part_obj: The json object representing this mesh part

All this method does is serialize the JSON object for the given mesh part to an ASCII string and then appends/writes that string at the end of the current file.

After serializing the object to an ASCII string and writing it to the file, the memory for the ASCII string is released by json_object_free_printbuf().

Return

A tiny JSON object holding the name of the file, the offset at which the JSON object for this part was written in the file and the part’s ID.

void main_dump(int argi, int argc, char **argv, json_object *main_obj, int dumpn, double dumpt)

Main MIF dump implementation for this plugin.

Parameters
  • argi: Command-line argument index at which first plugin-specific arg appears

  • argc: argc from main

  • argv: argv from main

  • main_obj: The main json object representing all data to be dumped

  • dumpn: The number/index of this dump. Each dump in a sequence gets a unique, monotone increasing index starting from 0

  • dumpt: The time to be associated with this dump (like a simulation’s time)

This is the function MACSio main calls to do the actual dump of data with this plugin.

It uses MACSIO_MIF twice; once for the main dump and a second time to create the root (or master) file. However, in the second use, the file count is set to 1. That means that the root file is effectively written using serial (e.g. non-parallel) I/O.

It is a useful exercise to ask how we might improve the implementation here to avoid writing the root file using serial I/O.

int register_this_interface()

Method to register this plugin with MACSio main.

Due to its use to initialize a file-scope, static const variable, this function winds up being called at load time (e.g. before main is even called).

Its purpose is to add key information about this plugin to MACSio’s global interface table.

Variables

char const *iface_name = "miftmpl"

Name of the interface this plugin uses

char const *iface_ext = "json"

Default file extension for files generated by this plugin

int json_as_html = 0

Use HTML output instead of raw ascii

int my_opt_one

Example of a static scope, plugin-specific variable to be set in process_args to control plugin behavior

int my_opt_two

Another example variable to control plugin behavior

char *my_opt_three_string

Another example variable to control plugin behavior

float my_opt_three_float

Another example variable to control plugin behavior

int const dummy = register_this_interface()

Dummy initializer to trigger register_this_interface by the loader.

This one statement is the only statement requiring compilation by a C++ compiler. That is because it involves initialization and non constant expressions (a function call in this case). This function call is guaranteed to occur during initialization (that is before even ‘main’ is called) and so will have the effect of populating the iface_map array merely by virtue of the fact that this code is linked with a main.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334

#include <json-cwx/json.h>

#include <macsio_clargs.h>
#include <macsio_iface.h>
#include <macsio_log.h>
#include <macsio_main.h>
#include <macsio_mif.h>
#include <macsio_utils.h>

#include <stdio.h>

#ifdef HAVE_MPI
#include <mpi.h>
#endif

/*!
\defgroup plugins Plugins
@{
*/

/*!
\addtogroup MIF Template
\brief A simple MIF Plugin Template
@{
*/

static char const *iface_name = "miftmpl"; /**< Name of the interface this plugin uses */
static char const *iface_ext = "json";     /**< Default file extension for files generated by this plugin */
static int json_as_html = 0;               /**< Use HTML output instead of raw ascii */
static int my_opt_one;                     /**< Example of a static scope, plugin-specific variable to be set in
                                                process_args to control plugin behavior */
static int my_opt_two;                     /**< Another example variable to control plugin behavior */
static char *my_opt_three_string;          /**< Another example variable to control plugin behavior */
static float my_opt_three_float;           /**< Another example variable to control plugin behavior */

/*!
\brief Process command-line arguments specific to this plugin

Uses MACSIO_CLARGS_ProcessCmdline() to do its work.

This example plugin is implemented to route command line arguments to memory locations
(e.g. static variables) here in the plugin.  Alternatively, a plugin can choose to
route the results of MACSIO_CLARGS_ProcessCmdline() to a JSON object. MACSio's main is
implemented that way.
*/
static int process_args(
    int argi,      /**< [in] Argument index of first argument that is specific to this plugin */
    int argc,      /**< [in] argc as passed into main */
    char *argv[]   /**< [in] argv as passed into main */
)
{
    /* Can use MACSIO_CLARGS_TOJSON here instead in which case pass the pointer to
       a json_object* as first arg and eliminate all the pointers to specific
       variables. The args will be returned as a json-c object. */
    const MACSIO_CLARGS_ArgvFlags_t argFlags = {MACSIO_CLARGS_WARN, MACSIO_CLARGS_TOMEM};

    MACSIO_CLARGS_ProcessCmdline(0, argFlags, argi, argc, argv,
        "--json_as_html", "",
            "Write files as HTML instead of raw ascii [false]",
            &json_as_html,
        "--my_opt_one", "",
            "Help message for my_opt_one which has no arguments. If present, local\n"
            "var my_opt_one will be assigned a value of 1 and a value of zero otherwise.",
            &my_opt_one,
        "--my_opt_two %d", MACSIO_CLARGS_NODEFAULT,
            "Help message for my_opt_two which has a single integer argument",
            &my_opt_two,
        "--my_opt_three %s %f", MACSIO_CLARGS_NODEFAULT,
            "Help message for my_opt_three which has a string argument and a float argument",
            &my_opt_three_string, &my_opt_three_float,
           MACSIO_CLARGS_END_OF_ARGS);

    return 0;
}

/*!
\brief CreateFile MIF Callback

This implments the MACSIO_MIF_CreateFile callback needed for a MIF mode plugin.

\return A void pointer to the plugin-specific file handle
*/
static void *CreateMyFile( 
    const char *fname,     /**< [in] Name of the MIF file to create */
    const char *nsname,    /**< [in] Name of the namespace within the file for caller should use. */
    void *userData         /**< [in] Optional plugin-specific user-defined data */
)
{
    FILE *file = fopen(fname, "w");
    return (void *) file;
}

/*!
\brief OpenFile MIF Callback

This implments the MACSIO_MIF_OpenFile callback needed for a MIF mode plugin.

\return A void pointer to the plugin-specific file handle
*/
static void *OpenMyFile(
    const char *fname,            /**< [in] Name of the MIF file to open */
    const char *nsname,           /**< [in] Name of the namespace within the file caller should use */
    MACSIO_MIF_ioFlags_t ioFlags, /**< [in] Various flags indicating behavior/options */
    void *userData                /**< [in] Optional plugin-specific user-defined data */
)
{
    FILE *file = fopen(fname, "a+");
    return (void *) file;
}

/*!
\brief CloseFile MIF Callback

This implments the MACSIO_CloseFile callback needed for a MIF mode plugin.
*/
static int CloseMyFile(
    void *file,      /**< [in] A void pointer to the plugin specific file handle */
    void *userData   /**< [in] Optional plugin specific user-defined data */
)
{
    return fclose((FILE*) file);
}

/*!
\brief Write a single mesh part to a MIF file

All this method does is serialize the JSON object for the given mesh
part to an ASCII string and then appends/writes that string at the
end of the current file.

After serializing the object to an ASCII string and writing it to the
file, the memory for the ASCII string is released by json_object_free_printbuf().

\return A tiny JSON object holding the name of the file, the offset at
which the JSON object for this part was written in the file and the part's ID.
*/
static json_object *write_mesh_part(
    FILE *myFile,          /**< [in] The file handle being used in a MIF dump */
    char const *fileName,  /**< [in] Name of the MIF file */
    json_object *part_obj  /**< [in] The json object representing this mesh part */
)
{
    json_object *part_info = json_object_new_object();

//#warning SOMEHOW SHOULD INCLUDE OFFSETS TO EACH VARIABLE
    /* Write the json mesh part object as an ascii string */
    fprintf(myFile, "%s\n", json_object_to_json_string_ext(part_obj, JSON_C_TO_STRING_PRETTY));
    json_object_free_printbuf(part_obj);

    /* Form the return 'value' holding the information on where to find this part */
    json_object_object_add(part_info, "partid",
//#warning CHANGE NAME OF KEY IN JSON TO PartID
        json_object_new_int(json_object_path_get_int(part_obj, "Mesh/ChunkID")));
    json_object_object_add(part_info, "file",
        json_object_new_string(fileName));
    json_object_object_add(part_info, "offset",
        json_object_new_double((double) ftello(myFile)));

    return part_info;
}

/*!
\brief Main MIF dump implementation for this plugin

This is the function MACSio main calls to do the actual dump of data with this plugin.

It uses \ref MACSIO_MIF twice; once for the main dump and a second time to create the
root (or master) file. However, in the second use, the file count is set to 1. That
means that the root file is effectively written using serial (e.g. non-parallel) I/O.

It is a useful exercise to ask how we might improve the implementation here to avoid
writing the root file using serial I/O.
*/
static void main_dump(
    int argi,               /**< [in] Command-line argument index at which first plugin-specific arg appears */
    int argc,               /**< [in] argc from main */
    char **argv,            /**< [in] argv from main */
    json_object *main_obj,  /**< [in] The main json object representing all data to be dumped */
    int dumpn,              /**< [in] The number/index of this dump. Each dump in a sequence gets a unique,
                                      monotone increasing index starting from 0 */
    double dumpt            /**< [in] The time to be associated with this dump (like a simulation's time) */
)
{
    int i, rank, numFiles;
    char fileName[256];
    FILE *myFile;
    MACSIO_MIF_ioFlags_t ioFlags = {MACSIO_MIF_WRITE,(unsigned int) JsonGetInt(main_obj,"clargs/exercise_scr")&0x1};
    MACSIO_MIF_baton_t *bat;
    json_object *parts;
    json_object *part_infos = json_object_new_array();

    /* process cl args */
    process_args(argi, argc, argv);

    /* ensure we're in MIF mode and determine the file count */
//#warning SIMPLIFY THIS LOGIC USING NEW JSON INTERFACE
    json_object *parfmode_obj = json_object_path_get_array(main_obj, "clargs/parallel_file_mode");
    if (parfmode_obj)
    {
        json_object *modestr = json_object_array_get_idx(parfmode_obj, 0);
        json_object *filecnt = json_object_array_get_idx(parfmode_obj, 1);
        if (!strcmp(json_object_get_string(modestr), "SIF"))
        {
            MACSIO_LOG_MSG(Die, ("miftmpl plugin cannot currently handle SIF mode"));
        }
        else
        {
            numFiles = json_object_get_int(filecnt);
        }
    }
    else
    {
        char const * modestr = json_object_path_get_string(main_obj, "clargs/parallel_file_mode");
        if (!strcmp(modestr, "SIF"))
        {
            MACSIO_LOG_MSG(Die, ("miftmpl plugin cannot currently handle SIF mode"));
        }
        else if (!strcmp(modestr, "MIFMAX"))
            numFiles = json_object_path_get_int(main_obj, "parallel/mpi_size");
        else if (!strcmp(modestr, "MIFAUTO"))
        {
            /* Call MACSio utility to determine optimal file count */
        }
    }

    bat = MACSIO_MIF_Init(numFiles, ioFlags, MACSIO_MAIN_Comm, 3,
        CreateMyFile, OpenMyFile, CloseMyFile, 0);

    rank = json_object_path_get_int(main_obj, "parallel/mpi_rank");

    /* Construct name for the silo file */
    sprintf(fileName, "%s_json_%05d_%03d.%s",
        json_object_path_get_string(main_obj, "clargs/filebase"),
        MACSIO_MIF_RankOfGroup(bat, rank),
        dumpn,
        json_object_path_get_string(main_obj, "clargs/fileext"));

    MACSIO_UTILS_RecordOutputFiles(dumpn, fileName);

    myFile = (FILE *) MACSIO_MIF_WaitForBaton(bat, fileName, 0);

    parts = json_object_path_get_array(main_obj, "problem/parts");
    for (i = 0; i < json_object_array_length(parts); i++)
    {
        json_object *this_part = json_object_array_get_idx(parts, i);
        json_object_array_add(part_infos, write_mesh_part(myFile, fileName, this_part));
    }

    /* Hand off the baton to the next processor. This winds up closing
     * the file so that the next processor that opens it can be assured
     * of getting a consistent and up to date view of the file's contents. */
    MACSIO_MIF_HandOffBaton(bat, myFile);

    /* We're done using MACSIO_MIF for these files, so finish it off */
    MACSIO_MIF_Finish(bat);

    /* Use MACSIO_MIF a second time to manage writing of the master/root
       file contents. This winds up being serial I/O but also means we
       never collect all info on all parts to any single processor. */
//#warning THERE IS A BETTER WAY TO DO USING LOOP OF NON-BLOCKING RECIEVES
    bat = MACSIO_MIF_Init(1, ioFlags, MACSIO_MAIN_Comm, 5,
        CreateMyFile, OpenMyFile, CloseMyFile, 0);

    /* Construct name for the silo file */
    sprintf(fileName, "%s_json_root_%03d.%s",
        json_object_path_get_string(main_obj, "clargs/filebase"),
        dumpn,
        json_object_path_get_string(main_obj, "clargs/fileext"));

    MACSIO_UTILS_RecordOutputFiles(dumpn, fileName);


    /* Wait for MACSIO_MIF to give this processor exclusive access */
    myFile = (FILE *) MACSIO_MIF_WaitForBaton(bat, fileName, 0);

//#warning FIX THE STRING THAT WE PRODUCE HERE SO ITS A SINGLE JSON ARRAY OBJECT
    /* This processor's work on the file is just to write its part_infos */
    fprintf(myFile, "%s\n", json_object_to_json_string_ext(part_infos, JSON_C_TO_STRING_PRETTY));

    MACSIO_MIF_HandOffBaton(bat, myFile);

    MACSIO_MIF_Finish(bat);

    /* decriment ref-count (and free) part_infos */
    json_object_put(part_infos);
}

/*!
\brief Method to register this plugin with MACSio main

Due to its use to initialize a file-scope, static const variable, this
function winds up being called at load time (e.g. before main is even called).

Its purpose is to add key information about this plugin to MACSio's global
interface table.
*/
static int register_this_interface()
{
    MACSIO_IFACE_Handle_t iface;

    if (strlen(iface_name) >= MACSIO_IFACE_MAX_NAME)
        MACSIO_LOG_MSG(Die, ("Interface name \"%s\" too long", iface_name));

    /* Populate information about this plugin */
    strcpy(iface.name, iface_name);
    strcpy(iface.ext, iface_ext);
    iface.dumpFunc = main_dump;
    iface.processArgsFunc = process_args;

    /* Register this plugin */
    if (!MACSIO_IFACE_Register(&iface))
        MACSIO_LOG_MSG(Die, ("Failed to register interface \"%s\"", iface_name));

    return 0;
}

/*!
\brief Dummy initializer to trigger register_this_interface by the loader

This one statement is the only statement requiring compilation by
a C++ compiler. That is because it involves initialization and non
constant expressions (a function call in this case). This function
call is guaranteed to occur during *initialization* (that is before
even 'main' is called) and so will have the effect of populating the
iface_map array merely by virtue of the fact that this code is linked
with a main.
*/
static int const dummy = register_this_interface();

/*!@}*/

/*!@}*/