Top-down
and bottom-up
What is the main focus of your work in the NFDIxCS project?
We believe it is important to view software as part of research data. While many NFDI consortia focus on traditional data, this project specifically addresses the question of how software, as a result and tool of scientific work, can be managed effectively and made available for long-term use. One concrete approach is the development of so-called research data management containers, which contain not only data but also the associated software—for example, for secure and reproducible access over longer periods of time.
What indicators are being developed at NFDIxCS to measure the scientific quality and reproducibility of research data?
We develop our indicators specifically for each subcommunity, as research data in computer science is very heterogeneous—ranging from EEG data in human-computer interaction to mathematical formulas. For this reason, there are no uniform standards; instead, we analyze which quality criteria are relevant in the respective subject contexts. The goal is more about artifact evaluation, such as defining clear requirements at conferences that so-called replication packages or accompanying data sets should meet in order to ensure reproducibility and scientific traceability.
What exactly is an artifact evaluation?
For conferences, there are so-called artifact evaluation tracks: papers are first reviewed, followed by the corresponding reproduction packages. This involves checking whether the results are reproducible and whether the artifacts are well documented and comprehensible. With NFDIxCS, we want to significantly simplify and automate this process. Nowadays, it is often difficult to bring authors and reviewers together or to run the software locally, especially when documentation is incomplete. Our goal is to enable reviewers to test the software directly in the cloud without any installation hurdles, making the entire process more transparent, efficient, and pleasant for everyone involved.
Anne Koziolek is dedicated to making the quality of research data in computer science measurable and sustainably improvable. (© Rabea Strauch, KIT)
Anne Koziolek
is a professor of computer science and software engineering at KIT and heads Task Area M2 on quality management at NFDIxCS. Her goal: to make research data measurable and sustainably high-quality. Using a top-down and bottom-up approach, her team develops tools and guidelines to improve the traceability, accessibility, and reusability of scientific artifacts.
“Research data management is often seen as something you do for others. And sure, it's about making results traceable and reusable. But especially with larger projects such as a dissertation, you also benefit yourself if you take a structured approach. It's just that you often only realize this too late.”
Anne Koziolek
That sounds a bit like the numerous documents are also being “filtered”? First the paper, then the artifact—so there are two stages of evaluation?
Yes, although the screening process only takes place in the first step—that is, when the paper itself is reviewed. It works the same way as at any conference or journal: not everything is accepted, only those papers that convince the reviewers. Only then the artefact comes into play. It is then evaluated additionally, but not again in terms of “accepted or rejected”, but rather: How well is it documented? Is it reusable? Can the results actually be reproduced? For this, there are badges, i.e. seals of quality, which are visible directly on the paper. Some conferences are now considering whether artifacts should be submitted together with the paper, but currently it is still mostly a two-step process: first the paper, then the artifact. This allows reviewers to focus on the scientific contribution first and only then on the technical verifiability.
The artifact evaluation platform is to be developed at NFDIxCS using a top-down-bottom-up approach. What exactly does that mean?
In general, top-down means that you first have a goal or an idea of where you want to go, and then break it down into concrete steps or requirements. Bottom-up works exactly the opposite way—you first look at what already exists, what ideas or processes are already in place, and build on that. And it often works best when you combine both—that is, set a rough goal, but at the same time remain open to what is already there, and then bring it all together step by step. In our project, especially in the area of quality management, we take exactly this approach. On the one hand, we consider what quality actually means in relation to research data—that's more top-down. On the other hand, we look at existing practices in the community, such as the Artifact Evaluation Tracks, and try to strengthen and better support them—that would be the bottom-up part.
What are your hopes for the future development of NFDIxCS, especially with regard to the visibility of computer science?
What I really want is for NFDIxCS to gain international visibility. We don't just want to develop something that works in Germany, we want to create solutions that also attract international attention—ideally to the extent that they become the standard for managing research data in computer science. For me, it's not about the NFDIxCS label, but about the solutions being so convincing and user-friendly that they establish themselves on their own—simply because they make research data management noticeably easier and more practical. My focus is clearly on the computer science community. Of course, many approaches can also add value in other disciplines—especially with reproduction packages—but my central goal is to first firmly anchor the topic in computer science.
What advice would you give to your younger self, particularly with regard to research data management?
I would probably advise myself to approach the topic more systematically—right from the start. During my dissertation, I went through a phase where I conducted many variations of my experiments with different parameters—and at some point, it was no longer entirely clear which configuration had produced which results. This naturally makes it difficult to draw reliable conclusions. At that time, I realized how important it is to document everything clearly—in other words, to really record: What were the conditions? What metadata is involved? If you do this consistently, you save yourself a lot of work later on—including for yourself. Research data management is often seen as something you do for others. And sure, it's about making results traceable and reusable. But especially with larger projects such as a dissertation, you also benefit yourself if you take a structured approach. It's just that you often only realize this too late.
You have already voted for this poll.
Opinion
Artifact evaluation should be integrated into the main paper review (one combined step), not handled separately.
1: Strongly disagree 5: Strongly Agree
Step 1: Software is treated as part of research data (not separate)
Step 2: Communities are heterogeneous → indicators must be tailored per subcommunity
(no one-size-fits-all)
Step 3: Conferences use a two-step process: paper review first, then artifact evaluation
(badges for documentation/reproducibility)
Step 4: NFDIxCS aims to make artifact evaluation easier via cloud-based testing
(no local installation)
Step 5: Long-term goal: solutions become internationally visible and self-adopting
(because they’re convincing and user-friendly)
