Data Preservation

Scientific data is a highly valuable asset for future generations and its preservation is not a trivial task. Ensuring that this information will be useful and reliable is the main goal for the HDF mapping project.

HDF files are structured, hierarchical data files used by NASA -among others institutions- to store large numeric datasets; Scientist use this data to feed their climate models and to research our environment. There are a variety of API to access and manipulate this data in its different versions, however from the data preservation stand point, data in different formats but with the same content is equal despite its representation.

In reality, agnostic data is a recurrent issue that brings concerns for long-term data accessibility. Imagine a distant future where most of the languages and frameworks we know are deprecated, HDF API may require an endless list of patches to be compatible with new operating systems and new wrappers around the new languages. This is a very possible scenario that could cost more than a couple lines of code.

The HDF mapping project addresses this conservancy problem by using metadata maps in XML format. These maps are human readable and can be parsed and used by future languages and frameworks. The HDF maps are more than a pointer-list document, they externalize information about HDF datasets at different levels, describe their content and provide information about how to extract and interpret them making this data accessible, platform independent and reliable for future generations