Creation of a system for automating repetitive actions performed on a computer

Gytis Medelis; Jovita Urnikienė

doi:10.56131/tmt.2023.2.2.160

Authors

Gytis Medelis Šiaulių valstybinė kolegija / Higher Education Institution, Lithuania; Graduate of the study program “Program Systems”
Jovita Urnikienė Šiaulių valstybinė kolegija / Higher Education Institution, Lithuania; Lecturer of the Department of Informatics Sciences

DOI:

https://doi.org/10.56131/tmt.2023.2.2.160

Keywords:

automating click operations, automatic actions system, automated optical character recognition, automatic clicker

Abstract

As technology advances and time saves time, more and more tasks are being automated, reducing the need for manual work. The process of automation is beneficial in computer-based work as well. This article analyzes automation systems for mouse clicks, their benefits, and their applications in various fields.

Project problem. The automated click systems do not provide flexible, reliable, and convenient algorithmic functions for the use of complex marketing systems, mobile games, or a broader market segment.

The objective of this work is to create a system that simulates mouse clicks, swipes, and text input, allowing for repetitive actions, creating and saving action execution algorithms.

The theoretical part discusses similar automated systems, their advantages and disadvantages, text and image recognition algorithms, and popular libraries. The compliance of the systems with the ISO/IEC 9126 standard is analyzed. The programming language chosen for development is C++, Qt framework, the Tesseract library is used for text recognition, while the OpenCV library is used for graphic objects.

The practical part presents the project of the created automated action system, the development process, the system itself, and the system's application possibilities. The system includes 10 control tools: text, color, and image recognition systems, mouse click, text input, and swipe simulation systems, automated action deletion, algorithm execution/stop, settings selection, menu collapse/expand, and system shutdown. The system allows for the creation of mouse click, swipe, and text input algorithms, defining conditions, creating action instructions, and sharing instructions on a forum. The system recognizes texts, colors, and images.