DATA COLLECTION METHODOLOGY

Our data is collected from end user devices running Android and iOS systems. All measurements are executed towards a CDN that has a large geographical footprint and hosts a significant part of the content that is being accessed by the users. This ensures our results are a good approximation of the user’s actual quality of experience.

All measurements must contain accurate location information using GPS or wi-fi geolocation methods. Measurements are considered only from the apps that have been approved by SpeedChecker. Submitted measurements are checked to see if they are within expected ranges and additional security precautions are implemented to ensure measurement data is not being manipulated.

The data collection process aims to deliver a single measurement sample from every device in our crowdsourcing system device pool and we strive to remove all duplicates. Due to privacy settings on some users phones we cannot reliably detect unique devices therefore some devices have contributed to more than 1 measurement into this dataset.

MEASUREMENT METHODOLOGY

The methodology is based on the concept of the ITU-T Q.3960 (2016) “Framework of Internet related performance measurements” and “Supplement 71 to ITU-T Q-series Recommendations”.

This test methodology aims at delivering an accurate measurement of the maximum bandwidth available over a given internet connection. This is achieved by transferring multiple parallel data streams over separate TCP connections within a predefined amount of time. The transferred data consists of randomly generated data with high entropy.

Before the speed test is started, we perform ping tests on all CDN networks. The CDN which performs the best for the particular connection is selected for the subsequent test phases (download, upload, TTFB).

We perform outlier detection on the ping samples with 95/5 filtering and report minimum ping and calculate jitter from the remaining samples. Jitter is calculated based on individual ping results as the difference in ping time between every two consecutive packets. Jitter is always expressed as a positive number, so the absolute value of the difference is used. The average jitter is defined as the average value of the jitter of consecutive packet pairs.

Due to CDN networks limitations we perform Packet loss to our measurement servers which are hosted at different datacenters than the CDN used for the rest of the tests. Packet loss is performed after the download / upload phases are completed (therefore is not run under load). The test is executed using UDP protocol. The measurement agent sends 1000 packets with 5ms delay to the measurement peer. The measurement peer acknowledges the packet reception back to the measurement agent. When the test completes, the measurement agent calculates the packet loss as a ratio of unconfirmed packets to total packets sent.

CONSISTENT QUALITY MEASUREMENT METHODOLOGY

Consistent quality is a KPI calculated from a selection of different measurements executed from the same user device in sequence. It is a valuable metric for determining whether the user connection satisfies the minimum thresholds for typical user activities.

We have three levels of Consistent quality:

Excellent

The network connection delivers the highest-quality user experience for popular video streaming sites, video games, video conferencing, or VOIP applications.

Core

The network connection supports core internet activities such as web browsing, using apps, email, SD video streaming etc.

Poor

The network connection does not meet thresholds for Core consistent quality.

Methodology

Consistent quality is calculated based on the following metrics:

• Download speed
• Upload Speed
• Ping
• Jitter
• Time to first byte (TTFB)
• Packet loss

Each consistent quality level has minimum thresholds for each of the measurements.

In order for a connection to be labelled as Excellent or Core, it must satisfy all six conditions for the measurements. If the conditions for Excellent are not met, we check if it meets the conditions for Core. If Core conditions are not met, we label the connection as Poor.

COVERAGE METHODOLOGY

SpeedChecker has developed a robust and reliable methodology of assessing cellular coverage worldwide. Our data-driven approach is using billions of cellular measurements conducted by hundreds of millions of mobile devices.

Process

SpeedChecker data analysis process for mobile coverage involves four primary steps: collection, filtering, spatial aggregation, and summarization. The results of that process are used to determine coverage score on a country and operator level.

Collection

Millions of cellular measurements are received daily from Android devices around the world.

Filtering

Filtering is applied to ensure that only relevant measurements are used:

• Erroneous out-of-range measurements are excluded from datasets
• Measurements executed by inactive devices are excluded
• Measurements with inaccurate location are excluded

Spatial Aggregation

Filtered multi-RAT signal measurements collected for last 12 months are grouped into tiles of approximately 1 km2. All tiles where at least one operator service was detected are summed up to form a total country coverage footprint. Average signal strength is calculated per tile per RAT for each operator. Each operator is then assigned with a score per tile depending on average signal strength and particular RAT availability (higher RAT and stronger signal will contribute a higher score).

Summarization

Finally, scores from all tiles are summed up per operator. This forms the total operator coverage score. Total operator coverage score is then divided by the total country coverage footprint to produce an overall OPERATOR COVERAGE SCORE.

*RAT – Radio Access Type