Euro-Par 2020 Call for Artifact Evaluation

The Euro-Par conference series encourages authors of accepted papers to participate in the Artifact Evaluation Process (AEP). The authors of papers accepted at Euro-Par 2021 will be formally invited to submit their support material (e.g., source code, tools, benchmarks, datasets, models) to the AEP to assess the reproducibility of the experimental results presented in the accepted paper. The artifact will undergo a completely independent review process, run by a separate committee of experts who will assess the quality of the artifact, the reproducibility of the experimental results shown in the paper, and the usefulness of the material and guidelines provided along with the artifact.

All artifacts will receive a review. The review will consist of few comments stating whether the evaluation was successful or not and providing hints for improving the document. A technical clarification window will occur, during which the reviewers can, anonymously, ask the corresponding authors of the artifact to solve technical issues encountered. The issues must be clarified within few days, otherwise the artifact will not be accepted.

The papers whose artifacts will be accepted, will receive a seal of approval printed on the first page of the papers as they appear in the final proceedings published by Springer. The artifact material will be made publicly available.

Although warmly advised, the artifact evaluation process is completely optional and, in any case, will not modify the acceptance decision already made on the Euro-Par papers.

The following methods are acceptable to make your artifacts available to the reviewers:

(A) A working copy of software (and dependencies), along with README files, datasets, examples, benchmarks and case studies needed to rebuild the software on the reviewers' machine and to produce the results contained in the accepted paper.

(B) A virtual machine image (Virtual Box, VMware) or Docker container, with all the necessary pre-installed software that allows to produce the results contained in the accepted paper.

(C) Arrange temporary remote access to the authors' machine with the pre-installed software - this option should be adopted when specific hardware is required or when there is proprietary software released under non-open-source licenses or that cannot be freely (and anonymously) downloadable. You will need to privately send the access information to the AEP chairs upon submission of the artifact.

All necessary packages, dependencies, and any additional software required to run the artifact must be explicitly listed in the Overview Document (see below) and (if possible) included in the artifact ZIP file.

If you adopt option A to make your artifacts available, the ideal target platform for evaluating the artifact should be a small cluster (1-3 nodes) of standard multicore servers equipped with one GPU and interconnected via a standard switched Ethernet network. The reference OS is Linux. Artifacts requiring specific non-commodity hardware not available to the Evaluation Committee, must use option C, or they will not be evaluated.

Overview Document for the Artifact

The Overview Document (that should be just a few pages) must contain all the exact steps to access, install, compile, and execute the artifact. Notably, the document must include comprehensive guidelines to assess the quality of the execution’s outcome and how to interpret the results with respect to the Euro-Par accepted paper.

Your overview document should consist of two parts or sections:

• a Getting Started Guide, and
• Step-by-Step Instructions on how to reproduce the results (with appropriate connections to the relevant sections of your paper).

The Getting Started Guide should contain setup instructions including the additional software to install with their exact versions and basic testing of your artifact. It is expected that this phase should require no more than 30 minutes to complete. You should write your Getting Started Guide to be as simple and straightforward as possible, and yet it should stress the key elements of your artifact. Hence, anyone who has successfully completed the Getting Started Guide should not have any technical difficulties with the rest of your artifact.

The Step-by-Step Instructionsshould explain how to reproduce any experiments or other activities that support the conclusions in your paper in full detail. Write this part in a way that it is useful for future researchers who have a deep interest in your work and want to compare with it or improve your results. In this section, you must specify the exact platform you have used for your tests and, for each input dataset that must be used to reproduce your experiments, the execution time it took on your system.

If reproducing your experiments takes several hours, please clearly state this at the beginning of the Step-by-Step section and clearly point out ways to run it on smaller inputs to reduce the execution time (yet obtaining qualitatively acceptable results). Artifacts requiring only long-running executions to produce meaningful results will not be evaluated.

Where appropriate, include descriptions of each test and link to files (included in the ZIP) that represent expected outputs, e.g., the log files expected to be generated by your tool on the given inputs, or expected results for each input file.

For performance experiments, if the artifact is to be executed outside the original testing environment (options A and B from Artifact Preparation Guidelines), it is understood that results will not perfectly match those in the paper, due to differences in the hardware. However, the artifact evaluators should be able to reproduce the same qualitative outcomes contained in the paper.

Where possible, please automate data extraction and the production of plots, so that wherever possible the experiments run using the artifact produce figures matching those figures in the paper.