Data is from Mars,
research is from Venus?

Does planetary data come with special requirements? 

Absolutely, particularly in terms of formats. We've always had a unique relationship with planetary data formats due to the limitations of Earth-based formats. Over time, we created several formats like the ISIS3 cube, Vicar, FITS, and of course the Planetary Data System (PDS) to meet our needs, the latter being widely used within the planetary domain. Our approach with PDS focuses on simplicity for longevity, ensuring that data can be easily accessed and used even 50 years into the future. An issue arose with formats like JPEG 2000, which, although standardized and widely used in the GIS community, are complex and require rigorous decompression software. PDS Version 4 has adopted a simpler method, using raw pixel streams with detached labels that describe the file format comprehensively. This makes the data easier to open and visualize without needing sophisticated software. We also ensure the use of simple table data formats like ASCII or binary, and we avoid complex data cube structures. Despite the simplicity, the PDS label is unique, making it necessary to support other formats inside a PDS archive carefully. Our challenge remains balancing simplicity for longevity with modern needs like data compression for streaming. Thus, while we maintain archives for the future, we may need to duplicate data to meet current technological demands.

What are the challenges you're facing with the planetary data system? 

While we're proud of our services, the usability of PDS has historically been a challenge, often falling short of the FAIR principles. The data, though meticulously archived and well-labeled, can be complex and difficult to use, especially for those not deeply familiar with the domain. This complexity is partly due to the diverse and unique formats we've had to create for planetary data, which differ from Earth-based formats. Additionally, the raw data from spacecraft often requires significant processing and calibration to be scientifically useful. Many missions are not funded to provide this additional processing, making the data hard to use directly for research. To address this, we've been developing the concept of a Planetary Data Ecosystem (PDE) to make data more accessible and usable for researchers. This involves creating Analysis Ready Data (ARD), which is pre-processed and calibrated for easier use. We've started implementing new standards like the Spatial Temporal Asset Catalog (STAC) to support this effort. Furthermore, we're working to ensure our data meets FAIR standards by improving metadata, ensuring proper documentation, and assigning DOIs to datasets. This effort aims to bridge the gap between raw data and derived products, making it easier for researchers to use the data without needing deep technical expertise in data processing. By enhancing the usability of our data, we support a wide range of research applications, from machine learning to geologic mapping and mineral resource analysis.

What tools or formats do you prefer for managing metadata, both when using metadata from other researchers and when uploading your own for other to see? 

As a federal employee working for the US government, I am mandated to support metadata for any product we produce. The PDS has its own metadata standard, which is necessary due to the broad scientific domain it covers. This includes astronomy, planetary surfaces, and various scientific disciplines. The PDS uses an XML standard with discipline dictionaries to describe different aspects of the data, such as cartography, imaging parameters, geometry, and spectral information. While this is a robust and comprehensive model, it can be complex and challenging to manage. To improve interoperability, we have been working on cross walking between different metadata standards. This involves developing tools that can translate metadata from one standard to another, ensuring that our data can be used effectively by a wider range of researchers. While it would be ideal to standardize on a single metadata format, the diversity of scientific domains in planetary research makes this impractical. Therefore, our focus is on developing methods to seamlessly convert and integrate metadata across different standards.

What are practices you have found to make a big difference in this context?

Supporting open-source projects has proven invaluable in advancing research. Open-source software, including libraries and tools on platforms like GitHub, often enhances both commercial and academic applications. By making sure that these resources are well-documented, licensed and accessible, we can significantly improve their usability and impact. For instance, using open-source tools and workflows, such as Jupyter notebooks, has facilitated better integration and accessibility of complex software. Transforming software from languages like C++ into more user-friendly formats like Python has been particularly beneficial for researchers.

What are the most significant difficulties you have faced in research data and software management over the past two decades?

The most significant challenge is making Analysis Ready Datasets (ARD) meet the needs of different scientific domains. For example, while raw data might be analysis-ready in fields like plasma or rings, surfical data requires extensive processing for mapping or spectral analysis. Different research focuses require different levels of data readiness. So, we have to provide both pre-processed data and at the same time enable scientists to access and reprocess original raw data.

What would be your recommendation to overcome these difficulties?

I would recommend to avoid reinventing the wheel. We should support existing formats and standards that are widely used and recognized by the community.

Is there a message to your younger self about preparing for the current challenges in research data management?

I would emphasize the importance of not just releasing data, but also ensuring it is well-documented. Even after 20 years, we're still revisiting old data to improve its quality and ensure it meets standards. I would advise against the fear of data abuse by encouraging good descriptions and clear processing steps. It's important to address concerns about data misuse by making datasets "fit for use"—clearly stating their accuracy, precision and limitations. Including a well-crafted metadata file or ReadMe can significantly improve the usability and credibility of the data. In the end, I could have learned a lot from adding better metadata.