The Evaluation of “APROMED” Application Based on Organizational Assesment

Developing an outpatient reservation application is one of the efforts to fulfil the requirements of PMK RI Number 21 of 2020. Kaliwates General Hospital has an outpatient reservation application called APROMED. However, 7.3 more times patients made reservations manually than using APROMED, and the decrement in APROMED users is 3.2–5.6% monthly. Personally, the informant wanted an APROMED usage level above 50%, but it was only 13.6%. Based on these data, the problem is the low usage level of APROMED. This research aimed to evaluate the APROMED application based on an organizational assessment at Kaliwates General Hospital because a thorough evaluation had never been done. The method used was descriptive research with the cross-sectional study by measuring based on the success components of the DeLone and McLean systems using the ISO/IEC 15504 measurement scale. The results obtained were 76.9% 'L' largely achieved for system quality value, 52% 'L' largely achieved for information quality, 73.9% 'L' largely achieved for service quality, 32% 'P' partially completed for user satisfaction, and 51.8% 'L' largely achieved for net benefits. Kaliwates General Hospital must support APROMED application development based on evaluation, research


INTRODUCTION
Based on the explanation of the Head of the IT Sub-division, the APROMED user does not achieve the target, so it cannot be evaluated based on terms of usage.
Nagameka and Anshori Research (2018) stated that the absence of achievement targets for a system impacts the implementation and use of the system, which cannot be effective.The informant wanted APROMED usage to be more than 50%, but it was only 13.6%.Various aspects can cause low-level usage of an application.
The APROMED application can be downloaded via Google Playstore so users can provide assessments and comments regarding the application.Most comments 91 Jurnal Ilmu dan Teknologi Kesehatan, Vol.11, No.1, September 2023, pp: 89-106 DOI: 10.32668/jitek.v11i1.1313stated that the application often experienced system errors, such as inability to open the application, forced-closed, or taking a long time to load.
One effort to respond to the problems is to hold a system evaluation.The informant stated that the organization had never carried out evaluations based on assessment or measurement of the system.System evaluation is important because it offers excellent opportunities to achieve organizational goals (Alhendawi & Baharudin, 2017).Evaluation helps improve system or application performance and determine the success of implementing information technology in the health sector (Purba, 2022).The system evaluation results will provide information on obstacles to online outpatient reservation services that may occur in the future (Nabyla & Sigitta, 2019).Potential problems faced by users and organizations can be identified by carrying out evaluations so that the development of the online patient reservation or registration system is better refined (Martiana, 2020).Gupta et al. (2007).
The measurement scale used the ISO/IEC 15504 standard.

RESULTS AND DISCUSSION
The Rolas Medika Patient Application, abbreviated as Kaliwates General Hospital, launched the APROMED application on February 2, 2022.This application aims to improve the quality of services in hospitals.
APROMED has three main functions: outpatient service reservations, a drug queue tracking system, and ambulance reservations.

Evaluation of the quality of the APROMED application system
Evaluation of the quality system of the APROMED application was assessed through five indicators: ease of use, flexibility, reliability, security and response time.convenience aspects, including ease of understanding, ease of learning and ease of use or operation.APROMED had no howto-use information as an introduction.The introduction shows the main menu before the system runs, providing the user with initial knowledge (Baharuddin et al., 2021).

Other research on the Online Dentist
Reservation System at RSGM, Padjadjaran University, stated that the application needs an initial display to direct users to access the system correctly (Suryani et al. 2021).
APROMED users also cannot reuse data that had been input because APROMED reservation data was disconnected.The system needs to implement features so users can use data without re-entering the same data again (Adikara et al., 2020).
Kaliwates General Hospital has also considered this aspect important to make APROMED easier for users.Based on evidence from meeting minutes regarding the evaluation of the user-friendliness of APROMED at Kaliwates General Hospital, it was stated that the APROMED menu was not user-friendly and needed an improvement for displaying doctor's schedule information.
Flexibility is another aspect not fulfilled in the system quality component.Largely Achieved (L).This value showed that most of the information quality components in the application had been fulfilled.However, re-measurements still need to be made after improvements to achieve the Fully Achieved (F) category (Muttaqin et al., 2020).In contrast to research conducted by Krisnawati et al. (2019) Regarding evaluating the hospital and ease of use.System accuracy evaluation must be carried out to maintain the system's consistency (Rahmad, 2020).
The next aspect is the aspect related to format (presentation).APROMED does not have design guidelines, and APROMED development does not refer to guidelines or standards.The design of an application should consider other people's perceptions, so standards are needed as a reference for fulfilling these perceptions (Firantoko et al., 2019).
The next aspect that has not been fulfilled is because APROMED does not have a proper display.Registration or reservation information flow as an image reduces patient complaints about service because it is easier to understand (Fitriyah, 2021).
APROMEDis also unavailable in multiple languages.The number of application users can increase if the application has multilingual features (Mulyaningtyas & Setyawan, 2021).Other aspects were not fulfilled because APROthe MED design evaluation was never done.Design is one of the main attractions of an application, so evaluation regarding design is important to carry out (Urbanantika & Salim, 2021).
Because the developer focused on improving the system but did not consider the APROMED design.

Evaluation of the quality of APROMED application services
Evaluation of the service quality of the APROMED application is assessed through three indicators, namely responsiveness, assurance, and empathy.

Evaluation of APROMED application user satisfaction
Evaluation of user satisfaction is not carried out directly on users but rather on confirmation by the hospital whether or not  The score obtained from measuring user satisfaction with the APROMED application is 32%, while 68% of aspects of user satisfaction have not been met.The user satisfaction value is included in the 'P' partially achieved assessment category because it is in the range >15-50%.This value shows evidence of the approach and some achievement of the aspects that must be met in the assessment process.In this category, several aspects of user However, regulation must still be carried out (Muslimah et al., 2023).
Many of these aspects have not been fulfilled because all aspects related to user satisfaction surveys have not been carried out by the hospital, which causes the user satisfaction survey scores low.User satisfaction surveys help determine the application's needs (Ani Yoraeni, 2021).
Applications that meet aspects according to their needs can reduce obstacles, where patient satisfaction using online reservations or registration will increase if the application does not frequently experience problems (Rohman et al., 2022).
User satisfaction can measure an application's success (Anggraini et al., 2020).Application user satisfaction surveys must be carried out to increase application users and obtain information regarding developments that must be implemented.
This has never been done directly with patients because many things need to be prepared, such as making questionnaires and others.Patient satisfaction surveys should be able to be carried out online in the APROMED application itself by displaying it at the end of the application after the patient uses the application.

Evaluate the net benefits of the APROMED application
The net benefits of the APROMED The score obtained from measuring the net benefits of the APROMED application was 51.8%, while 48.2% of aspects of information quality could not be met.The net benefits value is included in the 'L' assessment category as largely achieved (almost achieved) because the value is in the range >50-85%.This value shows evidence of a systematic approach and significant achievement of the aspects that must be fulfilled in the assessment process.
This category also explains that several weaknesses emerged when measuring the APROMED application in aspects of the net benefits component assessment process.This research has limitations in that evaluations should also be carried out at the registration or front office and at superiors (leaders) of hospital management at Kaliwates General Hospital Jember.Apart from that, the assessment is only seen from an organizational perspective and is not asked directly by APROMED application users or patients.In contrast, system quality indicators related to ease of use should be asked of users (patients) directly.

CONCLUSION
The overall evaluation of the APROMED stage so that several components cannot be fulfilled.To achieve a reasonably practical value, the value that must be achieved is in the range of 85-99%, and to achieve a practical system value, the value that must be achieved is more than equal to 100%.

Suggestions for Kaliwates General
Regulation of the Minister of Health of the Republic of Indonesia Number 21 of 2020 concerning the Strategic Plan of the Ministry of Health for 2020-2024 stated that hospitals required an effort to change health development governance, which includes integration of information systems, research and health development (PMK RI, 2020).One of the developments in information systems for health services is online outpatient reservations.Online reservations are helpful in helping patients obtain clinic information, doctors's operational hours, doctor's schedules, access to queue numbers, duration of procedures or examinations, and waiting time or queue information (Rizqi, 2018).Kaliwates General Hospital in Jember is one of the hospitals that has implemented an online reservation system by launching the Rolas Medika or APROMED Application.There is a problem, namely the low-level usage of the APROMED application.The number of outpatients from November 2021 to November 2022 was 128,275 patients.However, only 17,409 patients (13.6%) made reservations using APROMED, and 110,866 patients (86.4%) made reservations manually.This means 7.3 times more patients made management information system at Lawang Hospital, the value scored was 30%, including Partially Achieved (P).The quality of information is still not perfect because the implementation of SIM-RS at Lawang Hospital is only a basic application.However, the design of activities had not been adequately monitored and evaluated, and the work products have not been implemented, controlled and maintained as they should be(Krisnawati et al., 2019).On the other hand, APROMED provided up-to-date and complete information according to needs and had an APROMED usage module, namely a Guidebook for Using the APROMED Application for Patients.A video tutorial about how to use APROMED had been shared on the Kaliwates General Hospital YouTube channel.In the information quality component, the relevance aspect had not been fulfilled.APROMED did not provide the same information as manual reservations because it was not yet computerized.Adjustments were only made on a technical basis but not a system one.Based on research by Nabyla and Sigitta (2019), The design of the online hospital registration application for reservations or registration at a hospital must be made per the conditions for manual registration or reservation.This implementation was carried out by creating a flowchart describing the system's flow to be developed.This flowchart adapted the online patient reservation or registration system prototype to the conventional O. Miliani, D.N.G.W.M.Putra.The Evaluation of "APROMED" had not been fulfilled was related to accuracy.APROMED did not use standards to ensure the accuracy of the application, and there was no evaluation for accuracy.Following the previous discussion, APROMED development did not use any standards.Based on the interview results, APROMED had been evaluated only for system improvements they have carried out an evaluation such as a community satisfaction index (IKM) survey.Evaluation of user satisfaction (user satisfaction) of the APROMED Kaliwates General Hospital application is assessed through three indicators, namely repeat purchases (reuse), repeat visits (viewing or re-visiting) and user surveys.

99DOI
Jurnal Ilmu dan Teknologi Kesehatan,Vol.11,No.1, September 2023, pp: 89-106   .94% is  included in the 'L' category primarily achieved.Researchers stated that the results obtained were quite good.Only improvements and reevaluation were needed to achieve the best score for the user satisfaction component (Ramadhan & Tolle, 2022).This is similar to research conducted by Muslimah et al. (2023), who obtained an 'L' score primarily achieved with a percentage of 66% in evaluating the online registration system at RSUD Singaparna Medika Citrautama.The score was almost achieved because user satisfaction was evaluated with the system being considered user-friendliness.
application is less effective due to several obstacles, such as being in the development O. Miliani, D.N.G.W.M.Putra.The Evaluation of "APROMED" Hospital are to improve the system's quality by adding an introduction and carrying out developments so that APROMED can be installed on iOS and PC.APROMED needs to be synchronised by the system with manual reservations.Regarding the quality of information, it is necessary to add a visualisation of images or flows in the APROMED application, display posters of the flow of how to use APROMED in the outpatient waiting room and carry out evaluations regarding the system's accuracy.Improve the quality of APROMED services by adding a confirmation feature.The next suggestion is that Kaliwates General Hospital needs to support the development of the APROMED application through evaluation, research, surveys with application users to determine the evaluation of application use for users, namely patients, as well as conducting quantitative research based on the indicators described in this research to determine the relationship and the influence of each variable on the effectiveness of using APROMED.

Table 1 .
System quality measurement results

Table 2 .
Information system measurement results The first aspect was that APROMED had no confirmation feature.Online registration or O. Miliani, D.N.G.W.M.Putra.The Evaluation of "APROMED" Application Based on Organizational Assesment 94 DOI: 10.32668/jitek.v11i1.1313reservation applications should have a feature that can be confirmed automatically. of APROMED was not integrated with other systems such as SatuSehat or BPJS-Kesehatan.Based on the interview results, APROMED is still a stand-alone system that can`t synchronise with patient manual reservations.Research by Pradiatiningtyas and Kusnadi (2020) regarding the Design of an Outpatient Registration Information System for Patients at RSU PKU Muhammadiyah Bantul Yogyakarta stated that the outpatient registration or reservation system needs to implement a computerized system to increase the efficiency and effectiveness of the system.

Table 4 .
Results of user satisfaction measurements

Table 5 .
Net benefits measurement results