Usability Criteria

The usability of the systems was evaluated using 23 objective criteria which were divided into three groups of criteria related to the system workflow process, the presentation of the results, and additional aspects. The points were assigned based on researcher findings during a specific period of the time.

Workflow Process Usability

The first criteria group is related to the usability of the workflow process of using the systems. It was evaluated using the following questions:

1. Is it possible to upload and test multiple documents at the same time?
2. Does the system ask to fill in metadata for documents?
3. Does the system use original file names for the report?
4. Is there any word limit for the document testing?
5. Does the system display text in the chosen language only?
6. Can the system process large documents (for example, bachelor thesis)?

Result Presentation Usability

The presentation and understandability of the results reported by the systems were evaluated in a second usability criteria group. Since the systems cannot determine plagiarism, the results must be examined by one or more persons in order to determine if plagiarism is present and a sanction warranted. It must be necessary to download the result reports and to be able to locate them again in the system. Some systems rename the documents, assigning internal numbering to them, which makes it extremely difficult to find the report again. Many systems have different formats for online and downloadable reports. It would be useful for the report review if the system kept the original formatting and page numbers of the document being analyzed in order to ease the load of evaluation.

All criteria for this usability group are listed below: 

1. Reports are downloadable.
2. Results are saved in the user’s account and can be reviewed afterwards.
3. Matched passages are highlighted in the online report.
4. Matched passages are highlighted in the downloaded report (offline).
5. Evidence of similarity is demonstrated side-by-side with the source in the online report.
6. Evidence of similarity is demonstrated side-by-side with the source in the downloaded report.
7. Document formatting is not changed in the report.
8. Document page numbers are shown in the report.
9. The report is not spoiled by false positives.

Other Usability Aspects

Besides the workflow process and the result presentation, there are also other system usability aspects worth evaluating, specifically:

1. System costs are clearly stated in the system homepage.
2. Information about a free system trial version is advertised on the webpage.
3. The system can be integrated as an API to a learning management system.
4. The system can be integrated with the Moodle platform.
5. The system provides call support.
6. The call support is provided in English.
7. English is properly used on the website and reports.
8. There are no external advertisements.