Anti-Phishing Test Methodology
Operating system / Browser
Microsoft Windows; details of the exact version and architecture used will be given in the individual test reports. Please note that phishing tests can be carried out on the anti-phishing features built into individual browsers, without an additional security product, or on the anti-phishing measures provided by security products – hence the use of the term “browser/security product” in this document. The browser(s) used in security product tests will be specified in the individual test report.
Aim of the test
The test is intended to demonstrate how effective the participating browser/security products are at recognising and blocking phishing websites, and thus protecting the user from being defrauded by these sites.
Any computer user who does not feel completely confident of their own ability to recognise and avoid phishing attacks will benefit from using a security product/browser with effective phishing protection. Any computer enthusiast or professional who provides technical support for family, friends, colleagues or clients will also be concerned with installing or recommending products that provide phishing protection for their supported users.
Definition of the threat
A phishing site is a website that attempts to mimic a pre-existing, legitimate website, or which purports to be from a pre-existing, legitimate body such as a bank, and aims to obtain user credentials with a view to directly or indirectly defrauding the user or carrying out some other sort of crime.
One very common type of phishing attack involves sending out spam mails purporting to be from a bank, with a message to recipients that they need to log on to their Internet banking account for one reason or another. A hyperlink is provided in the mail, supposedly giving the victims easy access to their online accounts. In reality, the link leads to a fake copy of the bank’s login page. This will capture the user’s login credentials, which can then be used by the perpetrators of the scam to e.g. steal money from the victim’s account.
It should be noted that a phishing website does not affect the user’s computer or device in any way. Provided it uses standard technologies such as HTML, a phishing website can be effective on any type device, and is independent of operating system or browser.
There are numerous types of online fraud which are not counted as phishing and consequently not considered in this test. For example, fraudulent sites which encourage users to enter personal data under the guise of offering a new service (as opposed to mimicking an existing service).
Many web-based malware attacks use legitimate web servers to host the malware executables. Equally, it is possible for phishing attacks to host their webpages on the servers of reputable organisations which the perpetrators have managed to compromise. A phishing page should be recognised regardless of where it is hosted, though if a legitimate top-level domain is blocked, this would be regarded as a false positive. For example, if a phishing page is hosted under the URL www.lycos.com/user2035/personal/index.htm, this particular URL should be blocked, but blocking lycos.com (a legitimate domain) would be a false positive.
Scope of the test
The test is optional; vendors who have joined the main-test series can decide whether or not to participate. As noted above, phishing attacks commonly use links in spam mails to persuade users to visit the phishing websites. Our phishing-protection test is exclusively concerned with the ability of the browser/security product to identify the website itself as fraudulent and warn the user. The vector leading to each phishing URL, be it spam mail or links in other web pages, is not considered. Our Spam Protection Test will indicate to readers which security products are most effective at filtering out spam emails.
The test is carried out on identical test machines.
If Windows 8.x is used as the operating system, the test will be done using both the Desktop and Modern versions of the browser where available.
The operating system and browser(s) to be used in a test will be announced to participating vendors before the test begins. The browser used will be a popular mainstream one, which is supported by all the participating vendors.
Identical operating system and browser configurations are installed on all test computers. Any anti-phishing mechanisms within the OS are deactivated. For security-product tests, anti-phishing features in the browser are deactivated (obviously they are left active in browser tests). One browser/security product is then installed on each machine and updated.
All settings are left at their default values. The products have unrestricted cloud access throughout the test. Before the test proper is run, all products will be tested to ensure that they are correctly configured and functioning properly.
Sources and numbers of test cases
The phishing URLs used in the test are extracted from spam emails and collected from the web using a crawler.
A minimum of 100 phishing sites will be used, but possibly many more, depending on the duration of the test. For false-positive testing, at least 100 legitimate online banking websites are used.
Test procedure for browsers/security products
Phishing websites have very short lives and may be taken down only hours after they are put online. To ensure that as many phishing pages as possible can be tested while they are still active, we test all the phishing URLs we receive immediately; any that turn out to be inappropriate are excluded from the results. A URL may turn out to be unsuitable if it is not a genuine phishing site, is offline, is evidently a duplicate, or can be seen to be malfunctioning (e.g. error messages appear when the page is opened). Our automated test procedure feeds the test PCs one phishing URL, to which all machines then browse simultaneously (ensuring that the availability of the page is the same for all products). This is done in a way which replicates a user clicking on a link in real life (as opposed to an argument being passed directly to the browser). Screenshots are taken to indicate whether or not a phishing page has been blocked and/or a warning message displayed. Each test PC is then rebooted and reset to its original configuration before the next test case begins. This ensures a level playing field for each test case, and prevents any possible confusion between warning messages for one test case and those for a following test case.
To be deemed successful, a product must warn the user that a site is considered unsafe. It does not have to physically block the site completely. We note that some URL-blocking products will display a warning notice in the browser, with a darkened and inactive representation of the web page in the background; a button or link in the warning box will allow the user to proceed to the page. We accept this as protected – there are no “user-dependent” results in this test. The same principle applies with false-positive testing; a warning message that would allow the user to continue to the page by clicking the appropriate button or link still counts as a false positive if it appears for a harmless site.
As with many of our other tests, we check that products are not reaching high detection rates at the expense of a high rate of false positives. A false-positives test is carried out using a number of popular legitimate websites that ask for user credentials or personal information; there will be an emphasis on online banking sites worldwide. A single false positive from an online banking site is sufficient to downgrade a program’s rating.