MRG Effitas selected the best antiviruses for Android

MRG Effitas has conducted extensive testing of Android antiviruses. A series of tests called “Android 360 Degree Assessment Program” evaluates the protection of mobile antivirus in real-life scenarios.

In the tests was used the Genymotion emulator with Android 8.0.0 OS images that were up as for September and October 2020. This test environment allows covering a significant portion of the consumer devices on the market.

The laboratory specialists paid attention to the following aspects of the tested products.

Early detection of threats

The first scenario (Early Stage Detection) assessed the effectiveness of early-stage threat detection when test samples were copied to the test device’s SD card. The device has not yet been infected and the malicious files have been downloaded and ready to be installed.

A really good antivirus should detect threats as early as possible, preventing users from installing malware on their devices. In this test, the following steps were taken:

1. Preparing the test device included installing and initializing the antivirus application (accepting the license agreement, downloading the latest antivirus definitions, accepting all permission requests, etc.). Upon request antivirus was allowed to scan files on the SD card. If the vendor provided additional setup instructions, they were taken into account at this stage.
2. Setting up a mobile antivirus to scan files on the SD card.
3. Load the test set of samples to the SD card and start scanning.
4. The antivirus was configured to delete suspicious files.
5. Scanning was restarted until warnings ceased to appear or all suspicious files were deleted.
6. Counting the remaining samples.

Detection during the installation phase

In the second scenario (Detection During Installation), each individual test instance was installed in order to check the protection capacity of mobile antiviruses.

1. Using the Android Debug Bridge (adb) toolkit, a malicious application was installed on the device. After installation, the antivirus received information about the new application and launched detection mechanisms.
2. The anti-virus was given the necessary time to complete all scanning procedures.
3. A screenshot of the screen with the results of the check was created. Displaying a warning or alert was considered a successful test case. All logcat logs have been saved to an external device.
4. Using adb, the sample was uninstalled and then the next sample was installed.

Note: on Android devices, the installation of a malicious application does not always have undesirable consequences, unlike the first launch, when the malicious code is directly executed. Running a sample can be detrimental from a safety standpoint. After the first launch, the malware that requests the SYSTEM_ALERT_WINDOW permission is able to continuously display a screen to the user asking for access to the device administrator. In such cases, the user cannot get rid of the application because they do not have access to the launcher, application menu, or device settings to uninstall the application.

False positive test

To conduct a comprehensive assessment of the protection effectiveness of test takers, was used a limited set of legitimate sample. They were all downloaded from a well-known third-party app store, did not exhibit malicious behaviour, and did not ask for too many questionable permissions.

Test samples

Real threats from the network

In the tests was used a set of 156 malicious sampleы. All of them can be divided into the following categories:

• 7% – SMS payments (SMSs, SMS Payment). The application provides functions for sending SMS messages to premium rates. Most of the samples selected were capable of automatically sending SMS and requesting the SEND_SMS system permission, which led to direct financial losses for the victim.
• 47% are Trojans. Trojans are applications that offer a specific set of functions. The user has certain expectations for functionality, but in fact the application requests permissions that do not correspond to the described functions. A typical example is the Flashlight application, which can request access to contacts, location data, and data transfers.
• 21% – Spyware. Samples in this category sent information to external servers that can be used to track user behavior and habits. Most adware applications are considered spyware because they send the IMEI, phone number, manufacturer name and phone model, and other data to ad networks.
• 16% – Financial software/Internet banking (Financial/banking). This type of threat is aimed at stealing money. Typical financial malware checks your browser’s mobile banking session connection and might try to display a phishing website or pop-up. The user might think that the session data has suddenly been reset. Typically, such samples request SYSTEM_ALERT_WINDOW permission as well as permission to access the list of jobs.
• 8% – PNP (PUA). 5 The term Potentially Unwanted Applications (PUP) is used to describe applications that perform actions that do not correspond to the intent of the user. This category includes applications with aggressive ad modules that allow advertisers to track the activity of individual users. For many users, privacy is more important than functionality, and no feature can compensate for the sale of activity data over the Internet. Antiviruses should inform the user about the PUP installation.

Many designs can be classified into several categories at once. For example, a malicious application displays aggressive ads and at the same time gets access to the SEND_SMS permission to send messages to paid numbers. Figure 1 shows the distribution of test samples.

Threat simulation

Simulators are individual samples introduced into the testing process to test detection procedures. The simulators were created to simulate an attack model using a “third-party malware store” that provides users with modified versions of trusted applications with backdoors and malicious modules.

For testing purposes, 5 specially created simulator samples were used. It is important to emphasize that these samples were not actually collected. The samples implemented the well-known technique of exploiting the accessibility features of the Android API, which is used to read screen content, SMS tokens, payment details, and other sensitive information. Simulators were modified versions of legitimate Android apps that sent keystroke data, SMS content, passwords, and other data to an external web service over HTTP.

Samples to check for false positives

A set of 10 safe samples from third-party app stores was used to test for false positives. Apps asked different permissions and offered different functionality.

Tested antiviruses

The following mobile antiviruses took part in testing. In addition to solutions from well-known brands with a good reputation and rich history, in the test also participated products of smaller vendors with a small market share.

Since the Google Play app store hosts many antiviruses for Android, protection apps with a large number of downloads were given priority.

• AVG AntiVirus FREE for Android
• Avira Antivirus Security for Android
• Bitdefender Mobile Security for Android
• Comodo Mobile Security for Android
• ESET Mobile Security for Android
• Malwarebytes for Android
• McAfee Mobile Security for Android
• Norton Mobile Security for Android
• TAPI Labs Antivirus & Security for Android
• Zemana Mobile Antivirus for Android
• Zoner AntiVirus Free for Android

Test results

The following products demonstrated 99% (or higher) detection rates for malicious test suite samples that were free of PUPs. All of them have received the MRG Effitas certificate:

• Avira Antivirus Security for Android
• Bitdefender Mobile Security for Android
• ESET Mobile Security for Android
• Malwarebytes for Android
• Norton Mobile Security for Android

Medium detection rate (non-PUA)

The numbers were obtained by averaging the values obtained from early detection tests and detection when installed with a test set without PNP.

Overall detection rate (no PUA)

PUA detection levels

Detection rates of Trojan samples

Detection rates of banking samples

Detection rate of SMS threats

Mobile spyware detection rate

Simulated threats detection rate

Conclusions

Based on the testing results, several conclusions can be drawn regarding the effectiveness of the anti-virus engines and the samples used.

Vendor reputation and additional features

In Q3 2020 testing, most well-known major vendors performed perfect or near-perfect in detecting real threats. For users, this means a wide choice of products, with some of the leading solutions available for free. In the future, additional functions such as VPN, parental control, single subscription for PC and mobile devices, etc. may become an important priority of choice.

“Antivirus is another application”

Another conclusion from the test results: the detection of many mobile antiviruses depends to a large extent on the metadata of installed packages (hashes, developer certificates, etc.). This means that unlike the Windows environment, antivirus on Android cannot get an idea of the real activity of another application, because this behaviour is not consistent with the basic principles of Android security, because “antivirus is just another application.”

As a result, by running the newly installed sample, it was quite difficult to get rid of some of the test suite samples with real threats. Thus, timely and correctly displayed detection is an absolute must for mobile antivirus engines.

Detection mechanisms

Testing has confirmed that most mobile antiviruses use different detection methods before and after installation. This is due to access to a different set of metadata in the case of the file on the SD card and in the case of the installed application.

In addition, the general idea of the initial scan is an important aspect of security. It provides the ability to scan an apk file prior to installation and subsequent malicious consequences. Pre-scanning features should be more progressive in all mobile antivirus for Android (in recent tests, some antivirus did not provide this feature).

Simulator detection

Many antivirus engines that have been successful in detecting simulators in past tests, this time were able to perform detections based solely on the package signature attributes. This means that the successful detection was the result of a mechanism that was largely prone to false positives. Because of this, in previous rounds of testing, many products have had problems detecting simulator samples.

MRG Effitas has repeatedly claimed that the simulators used do not represent a realistic scenario for the anti-virus engine. However, real-world threat analyses show that the scenario presented has been in use for almost 10 years. The scheme where an attacker selects an existing Android application to perform covert spy activities has been known for a long time. The most famous campaign with this approach is APT Dark Caracal, analysed by Lookout.

Discovery alert

The Android 360 Assessment Program’s test history shows significant differences in the way users are notified. When it comes to successful detection, effective and clear notification to the user is an important part of both the effectiveness of the antivirus application and the overall user experience.

Tested applications used one of the following approaches:

1. Launches a separate screen with a red background and several lines of threat description.
2. Use the Android notification system to provide important notifications, usually displayed on the status bar.

Both approaches have their merits. The separate screen is hard to ignore, and Android notifications better match the OS user experience. Android provides many options for customizing notifications, so notifications can be accidentally missed. Most of the applications tested take the first approach.

In terms of wording and overall design, notifications can be improved in many applications. When using the Android notification system, the developer’s task is to provide the user with as much information as possible to make the right decision.

The responsible user must read and manipulate the text displayed on the screen, which presents a significant mental burden, especially for non-tech-savvy users. As a result, a well-designed interface can be critical for the user when making a decision (which ultimately makes devices more secure).

A little earlier German laboratory AV-Test selected 8 best antiviruses for Windows 10.