1. INTRODUCTION
Mobile phones have become one of the most significant electronic devices of the twenty first century. According to a 2009 survey 82% of Americans never leave home without their phones and 42% go as far as to say they ‘cannot live without’ their cell phone (Synovate, 2009). Because of the important role that mobile phones play in everyday life, mobile device proliferation has reached a global level. In 2009 global mobile phone use surpassed 4·1 billion subscribers (ITU, 2009).
One quickly emerging field of applications for mobile phones is location-based services (LBS). LBS are applications that deliver information to the user based on their real-time or historical geographic position, as well as the position of other users or nearby locations. Due to public safety concerns in the late 1990s, the Federal Communications Commission (FCC) issued the “E911 mandate” that required US cell phone carriers to implement the ability to determine the location of any mobile phone placing an emergency phone call within 50 to 300 metres, depending on the type of technology used (FCC, 2009). This mandate propelled the implementation of the technology that would eventually support commercial LBS to the forefront of the agenda of cellular providers in the United States.
Of all the positioning technologies considered to meet E911 requirements, GPS-based solutions provided the highest level of position accuracy, an estimated 3–5 metres under ideal conditions, necessary for many next-generation location-based applications such as real-time navigation (Kupper, Reference Kupper2005; Porcino, Reference Porcino2001). While certain US wireless carriers such as T-Mobile and AT&T initially chose network-based positioning technologies such as Uplink Time Difference of Arrival (U-TDOA) to meet E911 requirements, the growing popularity of commercial LBS has encouraged these carriers to implement GPS-based solutions on their network as well (GPS World, 2008). Because of the implementation of highly accurate positioning technologies and the shrinking costs of GPS chip integration with mobile phone hardware, the GPS-enabled mobile phone market is expected to quickly expand over the next three years with more than 550 million units shipping in 2012 (GPS World, 2008). In order to allow mobile software to take advantage of these capabilities, a new version of the standardized Java Micro Edition (Java ME) Location Application Programmer Interface (API) is being created that will accelerate the creation of next-generation Java location-aware software for mobile phones (Barbeau et al., Reference Barbeau, Labrador, Winters, Perez and Georggi2008). Programmatic access to position information is also available on many other major phone software development platforms, including Google's Android, Microsoft's Windows Mobile, and Apple's iPhone.
As a result of these new technologies, global demand for LBS is forecasted to dramatically increase over the next few years. According to recent market research, the North American population of GPS-enabled location-aware services subscribers is projected to grow from 500 000 users in 2006 to 20 million users in 2011 (ABI Research, 2006a), and world population growth expected to expand from 12 million in 2006 to 315 million by 2011 (ABI Research, 2006b). These new mobile applications will provide users with an unprecedented level of convenience, as many existing broadcast services such as traffic alerts will become location-aware and selectively delivered to the user only when they are highly relevant to the user's real-time position. The value of these “location-aware” applications will become substantial worldwide, with GPS-based devices and systems expected to generate yearly revenues of $240 billion by 2013 (ABI Research, 2006c)
With many mobile applications depending on real-time position data, it is critical to independently evaluate the accuracy of the GPS technology that will provide this information. Some past studies have used an “estimated horizontal accuracy uncertainty” value reported in metres by GPS-enabled mobile phones for each GPS fix to quantify GPS accuracy in particular environments (Aguilar et al., Reference Aguilar, Barbeau, Labrador, Perez, Perez and Winters2007a; Aguilar et al., Reference Aguilar, Barbeau, Perez, Labrador and Winters2007b). While this circular area, which should indicate the 1-sigma confidence level of the reported position (i.e., the true position should lie within this circular area at a probability of approximately 68%), is useful for a basic evaluation of accuracy, a more thorough empirical study is desired that evaluates reported positions against a ground truth value with a known location.
This paper presents the results from an empirical study of GPS data gathered using GPS-enabled mobile phones. The reported position data is compared against the known location of the device, providing an absolute measure of accuracy. This data is also compared against GPS data gathered by stand-alone GPS units at the same time and location. Furthermore, the “estimated horizontal accuracy uncertainty” value that is reported by the cell phone platform to the mobile application for each GPS fix is compared against the observed error for the respective GPS fix, thus indicating whether the estimated accuracy values provided by the phone are a reliable measure of the true accuracy of the GPS fix. Tests were conducted under both static conditions (outdoors and indoors) and dynamic conditions (outdoors only).
2. GPS TECHNOLOGY
The Global Positioning System's (GPS) consists of a constellation of 24 solar-powered satellites orbiting approximately 12 000 miles above the earth. GPS is operated and maintained by the US Department of Defense (DoD) in collaboration with several other government entities. A mobile GPS receiver, located on the earth's surface, uses radio signals broadcast from these satellites to determine the receiver's position, usually expressed as latitude and longitude, as well as additional positioning information. GPS has become the most widely used technology to provide location information for many different applications.
2.1. Assisted GPS
A popular technology that enhances GPS, especially in the area of mobile phone networks, is Assisted GPS (A-GPS). In A-GPS systems, useful information is provided by the cellular network that can aid the GPS receiver to calculate an accurate position more quickly. A-GPS uses information from more than one source while calculating device position.
A-GPS reduces the time required for signal acquisition by eliminating sections of the signal search space. One of these methods is to rebroadcast the ephemeris data from the base station directly to the phone, instead of requiring the phone to download the ephemeris data directly from the slower satellite link. Additionally, this ephemeris data can actually be predicted up to several hours in advance, eliminating the requirement of the base station to wait for the broadcast from the satellite. The receiver uses this information on satellite position, atmospheric interference, and the resulting Doppler Effect on the GPS signal in order to reduce the number of potential frequencies it must search when trying to acquire a first fix. So, only a few frequency offsets and their possible code-delays must be examined.
Code Division Multiple Access (CDMA)-based cellular networks such as Sprint and Verizon can also provide tighter integration between the mobile phone and cellular network because the CDMA network itself uses GPS as a time reference in its wireless communications. On these networks, positioning is generally divided into two categories: Mobile Station (MS) (i.e., cell phone)-based and MS-assisted. MS-based A-GPS calculates the final position information on the device itself, while MS-assisted simply provides the calculated pseudo-ranges from the device to a network-based server, which uses this along with other information to determine the final position data. MS-assisted A-GPS is typically utilized for E911 purposes, and, since it requires network resources for each position calculation, access to this positioning technology is typically restricted from use by mobile application developers. MS-based A-GPS is the positioning technology commonly used by mobile application developers since it provides rapid position recalculation onboard the mobile phone, and therefore the analysis in this paper focuses on MS-based A-GPS.
2.2. High Sensitivity GPS
A major limitation to GPS is the inability for receivers to determine their position indoors, underground, under dense tree coverage, between large buildings in “urban canyons,” and anywhere else where a solid object obstructs the view of the sky. While GPS provides reliable location information outdoors that can meet the E911 demands and provide LBS services, a traditional GPS receiver can only provide limited location information indoors. While A-GPS can be utilized in mobile phones, it usually does not provide any better results than traditional GPS for indoor signal reception (Van Diggelen, Reference Van Diggelen2002; Vittorini and Robinson, Reference Vittorini and Robinson2003).
Traditional GPS does not have the sensitivity required to detect highly attenuated signals due to its design. When a GPS unit is activated, it begins to search through the space defined by frequency offsets and possible code-delays. GPS must assume the worst case scenario in order to provide reliable service, which is that the last searched frequency and code is where it will actually find a match to the signal. In order to reach this state in a reasonable amount of time (and therefore make sure the user isn't waiting an unreasonable amount of time for a result), the search time, or “dwell time,” per search space unit, or “bin,” is limited to one millisecond in a traditional unassisted GPS unit. If a match isn't found in this dwell time, the search moves to the next bin. This process continues sequentially throughout the entire search space until a signal is found or the last bin is searched and found to be empty. If a severely attenuated signal exists in a search bin but isn't found in the dwell time, then a GPS signal will not be found by the receiver. A-GPS receivers are provided information from the cellular network that reduces the necessary search space. Therefore, dwell time can increase and the user's wait time will remain the same. However, A-GPS typically allows only 10 times the traditional dwell time of standard GPS receivers, which still isn't sufficient for consistent indoor operation. The only way to increase a GPS receiver's sensitivity to the level necessary for indoor operation is to significantly increase the dwell time beyond what is possible with A-GPS, which is the catalyst for a new GPS architecture design. “Massively parallel correlation” is the term used to describe the new method of searching for a GPS signal indoors. Instead of processing the search space sequentially, all code delay possibilities for a satellite are searched in parallel. This allows a dwell time on the order of 1 to 10 seconds, since the dwell time occurs simultaneously for all code delay bins. This dwell time is sufficient to acquire a GPS fix in highly attenuated environments, including indoors (Van Diggelen, Reference Van Diggelen2002). A sequential search that had a dwell time of 10 seconds would require (10 seconds) X (1023 code delay possibilities), or 2·8 hours to cover the search space, even using A-GPS to eliminate much of the original search space. Obviously, the sequential search is not feasible for real-time use. Only a parallel search will make indoor GPS positioning possible.
High-sensitivity GPS chipsets have become widely employed in recent years and have been shown to perform quite well under challenging conditions (MacGougan et al., Reference MacGougan, Lachapelle, Klukas, Siu, Garin, Shewfelt and Cox2002; Schön and Bielenberg, Reference Schön and Bielenberg2008; Watson, Reference Watson2005). There are some concerns, however, that the improved availability of position fixes under these challenging conditions comes at the cost of reduced positional accuracy (Wieser, Reference Wieser2006). Despite these concerns, high-sensitivity GPS chipsets are currently very common in a range of different GPS-enabled devices. State-of-the-art location-aware mobile phones include both high-sensitivity GPS hardware and A-GPS capabilities. Custom hardware using over tens of thousands of correlators is implemented in silicon in order to quickly process and handle the massively parallel search of all code delays simultaneously while assistance information is retrieved from the network to aid in acquiring the fix. The end result is a GPS-enabled mobile phone that is capable of calculating a position where previous mobile phones could not.
2.3. Accuracy of GPS and A-GPS
The positional accuracy of a GPS position fix is influenced by a number of factors, including ephemeris error, ionospheric and tropospheric delays, receiver noise, satellite geometry and multipath effects. Without any form of correction, under good conditions autonomous GPS can achieve a horizontal positional error in the order to several metres. This will vary with receiver types and specific testing conditions. For example, Wing et al. (Reference Wing, Eklund and Kellog2005) tested six different consumer grade units and performance was found to vary considerably among units and was strongly influenced by canopy cover. Positional accuracy of the best performing units varied between 1 and 5 metres (average error) under open sky conditions. In contrast, commercial grade receivers with differential correction typically result in a positional error of less than 1 metre (e.g. Serr et al., Reference Serr, Windholz and Weber2006).
Differential correction methods exist to improve upon the positional accuracy of autonomous GPS, including the use of radio beacons and the Wide Area Augmentation System (WAAS) for real-time correction, and various forms of post-processing. These correction techniques can result in substantial improvement, but are typically not implemented as a direct observation by the handset to improve the performance of A-GPS for mobile phone software applications. However, some correction information from reference stations in the network may be provided to the handset in the assistance data sent by the network and therefore is used by the device to calculate the final position information. As a result of the tighter integration between the cell phone and network, CDMA networks also support hybrid positioning techniques where cell phone tower broadcasts can be used as a substitute for GPS signals to calculate course position data in highly obstructed environments when the receiver cannot see enough satellites to compute a reliable position (e.g. deep indoors, basements, etc.). These position calculations are classified under the MS-assisted category and are not typically exposed to general application developers for use in applications, or the frequency of position recalculations using this method are highly restricted by the network due to the extra load placed on network resources. Therefore, this positioning method is typically only used by the cell phone carrier for E911 purposes when necessary, or for a rough indoor position for a mobile application that does not require high accuracy (e.g., a local search or weather application). The focus of the analysis in this paper is MS-based autonomous and Assisted GPS accuracy since these positioning technologies are used by mobile applications when a high degree of accuracy is important and can be most easily compared over different cell phone models running on different wireless carrier networks.
While the accuracy of regular GPS has been very well documented for a range of different devices, A-GPS has not received the same attention. In general, A-GPS is expected to follow the same generic patterns as regular GPS in terms of the factors which affect its accuracy, such as the satellite geometry, receiver noise, etc. A few differences, however, can be expected. First, because of the assistance data provided by the network, A-GPS is expected to have a shorter Time to First Fix (TTFF). Second, in the design of mobile phones typically some concessions have to be made in terms of GPS hardware choices, especially the antenna. GPS hardware on mobile phones is competing for space and battery power, in contrast to regular GPS devices. While GPS chipsets are getting smaller, one of the distinguishing features of higher grade GPS units is a solid antenna system, which is often lacking on mobile phones. Based on these hardware considerations, A-GPS on mobile phones is not expected to perform as well as regular autonomous GPS receivers when it comes to positional accuracy.
In one of the few recent empirical studies on the subject, Zandbergen (Reference Zandbergen2009) compared the performance of A-GPS on Apple's 3G iPhone to regular GPS receivers. Results indicated that the typical median error for A-GPS locations was around 8 metres, while the typical median for regular GPS receivers under the same testing conditions was 1 to 2 metres. However, the largest error encountered during several static outdoor tests did not exceed 30 metres, confirming the utility of A-GPS locations for LBS applications.
3. METHODS
The general research design relied on a comparison of position fixes from GPS-enabled mobile phones at reference locations with known coordinates. Data was collected in three modes: static outdoor, static indoor and dynamic outdoor. For comparison position fixes were also collected at the same time using autonomous handheld GPS units.
3.1. Equipment
In order to collect Assisted GPS data from GPS-enabled mobile phones, a custom Java ME application utilizing the JSR179 Location API 1.0 was utilized. Using the LocationListener defined by the API, the application requested a position update interval of two seconds. Upon every location update, the Java ME application immediately transferred the location data wirelessly back to a server using the User Datagram Protocol (UDP) where it was stored for later processing and retrieval.
The GPS-enabled mobile phones used in these experiments were the Sanyo SCP-7050 running on the Sprint-Nextel CDMA 1×RTT data network and the Motorola i580 on the Sprint-Nextel iDEN network. The Sanyo SCP-7050 utilizes Qualcomm's MSM6125 chipset, which features a high-sensitivity −159 dBm GPS receiver as well as Assisted GPS. The Motorola i580 features a proprietary Motorola chipset which includes SiRF GPS hardware, and is also capable of Assisted GPS. The exact sensitivity of the GPS hardware in the i580 is not officially documented.
Both mobile phones recorded the position in decimal degrees using a WGS 1984 datum. In addition to the position information, the following variables were recorded for every position: altitude (in metres), speed, time and date, session ID, positioning mode (MS-based Assisted GPS, autonomous GPS or cell ID), and estimated horizontal and vertical accuracy (in metres).
Two low-cost handheld GPS units were used as a comparison: a Garmin GPSMAP 76 and a Trimble Juno ST. These units represent relatively low cost units with chipsets that are commonly used in various types of consumer-grade GPS devices. The Juno ST unit is equipped with a high-sensitivity GPS chipset, while the Garmin unit is equipped with a regular GPS chipset (i.e. not a high-sensitivity). Both units were used in autonomous mode only, and no WAAS or other forms of differential correction were applied.
3.2. Field Test Design
Three different types of tests were carried out: static outdoor, static indoor and dynamic outdoor. Each will be described in some detail.
In the static outdoor test, mobile phones and GPS units were mounted atop a survey tripod. The tripod was placed directly on top of a surveyed benchmark location of 1st order horizontal and vertical accuracy, as reported by the datasheet from the National Geodetic Survey. The units were placed on a 1 square foot plastic plate, mounted on the top of the survey tripod. The mobile phones and Juno ST units were placed horizontal on the plate while the Garmin units were placed upright, consistent with the manufacturers' recommendations for optimal antenna positioning. For both mobile phones the antennas were extended. All units were turned on and given sufficient time to establish a reliable GPS position fix. Logging rate was set to 2 seconds since preliminary testing revealed that this was the fastest rate that could be achieved across all units for reliable logging. All units were set to log simultaneously for approximately 1 hour and 15 minutes. In the data processing the start and end of the data logs were removed from the analysis to obtain a log record of exactly 3,600 seconds with the exact same start and end time for all units.
In the static indoor test, mobile phones and GPS units were placed in very close proximity on top of a regular wooden desk on the second floor of a two-story residential structure. The second floor of the structure consisted of a wood frame with cement stucco while the roof consisted of a wood frame with asphalt shingles. While GPS signal reception within this structure is possible (even without using a high-sensitivity chipset), the reception was severely affected by the building materials, resulting in lower expected accuracy. The data logging was identical to the static outdoor test, i.e. simultaneous 2-second logging for one hour.
In the dynamic outdoor test, mobile phones were mounted on a vehicle using a roof carrier with clamps to hold the units. GPS units were placed inside the vehicle and connected to magnetic patch antennas which were placed on the vehicle's roof. Mobile phones and patch antennas were placed near the centre of the vehicle. Units were turned on and given sufficient time to obtain a reliable position fix. Logging was started and a test track was driven. The test track in this case consisted of a cloverleaf intersection between Interstate 4 and US Highway 301. The actual driving lane(s) were determined in the planning phase using high resolution ortho-imagery. In the field test the vehicle was driven at approximately 25 miles per hour following the centreline of the driving lane as closely as possible. The test track was driven 10 times in sequence, taking approximately 23 minutes total.
3.3. Data Processing
Data logs from the mobile phones were checked for consistency. Specifically, for each position fix a determination was made whether the location was derived using GPS or if the reported position was a cell tower location – only the locations derived using GPS were used in further analysis. The percentage of valid GPS fixes was determined as one measure of the reliability of the mobile phones. Data logs from the GPS units were also checked for consistency, including missing data, invalid speed measurements, etc. No invalid fixes were identified in the GPS unit data logs for any of the tests. All raw data from the mobile phones and GPS units was recorded as latitude and longitude in the default WGS 1984 datum. In order to compare the locations with reference sites the data was projected into UTM Zone 17N NAD 1983 using the appropriate datum transformation.
3.4. Accuracy statistics
For the static outdoor tests both horizontal and vertical accuracy statistics were determined. Horizontal accuracy was determined as the Euclidean distance in metres in the UTM coordinate system between observed positions and the known coordinates of the surveyed benchmark. Vertical accuracy was determined as the difference between the observed altitude and the known altitude of the surveyed benchmark, corrected for the height of the survey tripod on which the units were placed. Horizontal and vertical accuracy were determined for every valid position fix, for a maximum of 1800 position fixes in a single test. The distribution of error values for every test was characterized by determining the following statistics: minimum, maximum, average, 50th, 68th and 95th percentiles and Root Mean Square Error (RMSE). The results for altitude for the Sanyo phone turned out to be very unreliable, as confirmed by the cellular carrier, and were excluded from the analysis.
For the static indoor test only horizontal accuracy was determined. The reference location was determined by marking the location on a blueprint of the residential structure. This blueprint was scanned and geo-referenced against a digital 6-inch resolution colour ortho-photo from 2006. The ortho-photo was already in the UTM Zone 17N NAD 1983 coordinate system, allowing for a direct determination of the UTM coordinates of the test location inside the residential structure. Horizontal accuracy was determined as the Euclidean distance in metres in the UTM coordinate system between observed position and the coordinates of the test location derived from the ortho-imagery.
For the dynamic outdoor test only horizontal accuracy was determined. The reference location was determined as the centreline of the driving lane. This centreline was obtained by digitizing a polyline in Arc GIS using digital 6-inch resolution colour ortho-imagery from 2006 as the reference. Horizontal accuracy was determined as the shortest Euclidean distance in metres in the UTM coordinate system between observed position and the polyline representing the centreline. This in effect determines the displacement perpendicular to the centreline, but not the displacement in the driving direction, since the true location of the vehicle along the centreline at a particular point in time is not known. Since the test track consisted of a cloverleaf intersection, data points were manually identified as belonging to an overpass or underpass to allow for a reliable error estimate. For example, points associated with an underpass could potentially fall closer to the polyline segment of the overpass, resulting in an underestimate of the positional error, but the manual correction eliminated this potential bias.
Further analysis was carried out to compare the observed positional error with the estimated horizontal accuracy provided with each position fix of the mobile phone. Scatter plots were created to provide descriptive comparisons and the number of times the positional error was over or under-estimated were determined for each test.
4. RESULTS AND DISCUSSION
4.1. Static outdoor tests
To illustrate the general nature of the data collected, Figure 1 shows the result of one of the static outdoor tests. For each of the four different units the scatter plot shows the individual position fixes collected over exactly one hour relative to the reference location. The circular buffers show distances of 5, 10, 15 and 20 metres around the surveyed benchmark.
Sample scatter plots of static outdoor accuracy tests.
The first general observation from Figure 1 is that the position fixes from both mobile phones reveal a strong gridded pattern, while no such gridding is evident for the regular GPS units. This implies that the precision of the coordinates being collected is not as high as for the regular GPS units. For the Motorola i580 the precision in east-west direction varies from 1·47 to 2·26 m while the precision in north-south direction varies from 1·00 to 1·55 m. For the Sanyo 7050 the precision in east-west direction varies from 0·69 to 0·78 m in east-west direction and from 0·33 to 0·65 m in north-south direction. These relatively large values for coordinate precision are a result of rounding in the processing of the GPS data, perhaps due to floating point precision limitations in the underlying mobile phone hardware or software. Since the exact nature of the processing is not revealed by the network providers or mobile phone manufacturers, it is unclear exactly how the rounding is introduced. The magnitude of the observed values, however, is approximately equivalent to reporting a position in decimal degrees using only five decimals, as is common is some older or low-cost handheld GPS units. Since the majority of hardware in a mobile phone chipset is dedicated to functionality other than GPS, space, energy consumption, and cost constraints likely limit the GPS processing capabilities of current mobile phones to levels that resemble older or low-cost stand-alone GPS units. The coordinates of the two GPS units used in the study are reported in decimal degrees using seven decimals, resulting in much higher precision. As a result of the gridding, the scatter plots of the two mobile phones looks much less dense since many observations have identical coordinates.
The second observation from Figure 1 is that the positions of the two mobile phones are much more spread out compared to those of the two GPS units. While all positions of the two GPS units fall within the 5 metre buffer around the surveyed benchmark, positions of the mobile phones commonly fall outside of the 10 and 15 metre buffers. This provides a general indication that the results for the mobile phones are much less accurate compared to regular GPS units. The scatter plots, however, do not reveal any systematic bias relative to the benchmark location, i.e. the points are scattered in a relatively random pattern. This suggests that A-GPS behaves like regular GPS in that the bias in observations over longer time periods is relatively small or absent. This is meaningful since one of the fundamental strategies employed in regular GPS to increase positional accuracy is averaging. For example, a waypoint would be collected by using the average of a number of positions collected over several minutes. The waypoint based on the average is expected to be more accurate with a larger number of positions. Results from Figure 1 suggest that such averaging could also be employed in A-GPS to improve positional accuracy.
The complete results for the static outdoor test are presented in Table 1. The Motorola i580 phone was used in a total of four tests, all in assisted mode. The Sanyo 7050 phone was used in four tests in assisted mode and in two tests in autonomous mode. Both GPS units were used in two tests in autonomous mode. The log record for all tests consisted of exactly 3,600 seconds. At 2-second logging this results in a maximum sample of 1,800 fixes. For GPS units, the percentage of valid fixes in Table 1 is determined as the number of valid fixes in each test relative to this maximum of 1,800. All GPS tests resulted in 100% valid fixes. For mobile phones, several communication and software issues were experienced which periodically interrupted the transfer of the location data from the mobile phone to the server. These issues effectively reduced the collected sample size for several of the mobile phone tests. Because of these issues, the percentage of valid fixes for mobile phones is determined by the number of GPS fixes received by the server in each test relative to the total number of location data transmissions received by the server.
Horizontal and vertical error statistics for static outdoor tests.
The RMSE values for the GPS units vary from 1·52 to 4·80 m, while these values for the mobile phones vary from 5·98 to 12·50 m. An independent samples t-test confirmed that the mean RMSE for the four GPS tests and the 10 mobile tests are significantly different (p<0·001). The RMSE values for the Motorola phone appear to be somewhat higher than for the Sanyo phone but an independent sample t-test and non-parametric Mann-Whitney U test did not reveal a significant difference. There was also no significant difference in the RMSE values for the Sanyo phone in assisted or autonomous mode.
While the RMSE statistic is commonly used to characterize the accuracy of GPS units, of particular relevance for GPS-enabled mobile phones is the 95th percentile since this is the statistic employed by the FCC in setting standards for the E911 requirements. The specific standard states: “for network-based solutions: 100 metres for 67% of calls, 300 metres for 95% of calls; for handset-based solutions: 50 metres for 67% of calls, 150 metres for 95% of calls” (FCC, 2000). The 95th percentiles of the error distribution for the mobile phones tested varied from 10·26 to 23·90 m. These results under static outdoor conditions easily meet the stated requirements. So while the horizontal positional accuracy of Assisted GPS position fixes from mobile phones is substantially less than those from regular handheld autonomous GPS units, the performance is within the current requirements.
Vertical accuracy of GPS positions is typically less than horizontal accuracy due to satellite configurations, typically by a factor of 1·5 to 3. This is evidenced in the results for the two GPS units where the value for RMSE and the 95th percentile are higher for vertical than for horizontal accuracy for most tests. The results for the Motorola phone show a similar pattern with values for RMSE and the 95th percentile roughly two to three times higher for vertical accuracy compared to horizontal accuracy. Altitude determination from the Sanyo phone was deemed unreliable and is not included. There are at present no E911 requirements for vertical accuracy of position fixes of mobile phones.
4.2. Dynamic outdoor tests
The results of the dynamic outdoor tests are shown in Figure 2 and Table 2. Figure 2 shows a close-up of a section of the test track and a more general view of the cloverleaf intersection as an inset map. Results show a clustering of the data points along the travel path, with a number of outlying points, in particular for the Motorola phone. In terms of percentage of valid fixes, both GPS units achieved 100% (despite the presence of underpasses), while the mobile phones achieved nearly 100% valid fixes. Horizontal accuracy statistics showed nearly identical results for RMSE and 95th percentile for the two GPS units, with slightly higher values for the Sanyo phone and substantially higher values for the Motorola phone. For all units the horizontal error values are lower than reported in Table 1 for the static outdoor tests. This may appear somewhat surprising initially, but the explanation lies in how positional error is determined. For the dynamic tests, the error is determined as the shortest distance to the travel path since the actual location along the path cannot be determined with sufficient accuracy. As a result, the error estimates in Table 2 represent the horizontal error in a single direction (perpendicular to the polyline representing the travel path) which is theoretically smaller than the horizontal direction in 2D space by a factor of approximately 1·41. Still, given this factor, the RMSE values are quite low relative to the static outdoor tests, suggesting that for a unit in motion the horizontal accuracy is not affected. This observation is important for many cell phone applications that depend on GPS position information while moving, such as real-time navigation applications, since it appears that the effect of moving at regular travelling speeds on the accuracy of GPS position information is limited.
Sample of position fixes during outdoor dynamic accuracy tests.
Horizontal error statistics for dynamic outdoor tests.
4.3. Static indoor tests
The indoor performance of GPS and A-GPS is expected to be inferior to outdoor performance due to signal obstruction. The results of the static indoor tests are shown in Figure 3 and Table 3. Figure 3 shows the scatter plots of the position fixes. Due to substantial differences in the magnitude of the errors, the image on the left shows the results for the mobile phones at a smaller scale than the image on the right for the GPS units. In terms of percentage of valid fixes, the high-sensitivity Juno ST unit achieved 100%, while the regular Garmin unit achieved 17·7%. This is to be expected given the nature of the GPS chipsets and the building materials. The Motorola and Sanyo mobile phones achieved nearly 100% valid GPS fixes due to the high-sensitivity nature of the GPS chipsets. The excellent availability of valid GPS positioning fixes indoors is obviously very encouraging since poor performance indoors is often quoted as a major weakness of GPS. It should be noted, however, that the indoor location was a residential building consisting mostly of wood and concrete stucco. Availability of sufficiently strong GPS signals in buildings with more concrete and steel is likely to be significantly reduced and even high sensitivity GPS chipsets are not expected to achieve consistent valid GPS fixes.
Scatter plots of static indoor accuracy tests.
Horizontal error statistics for static indoor tests.
The horizontal accuracy of all units indoors was less than for the static outdoor conditions. The Juno ST unit performed the best with a RMSE value of 6·16 m, followed by the Garmin unit with a value of 10·61 m. The Sanyo performed only slightly worse than the Garmin with an RMSE value of 11·33 m (and had a much higher percentage of valid fixes). This clearly indicates the potential of high-sensitivity GPS chipsets in mobile phones to perform reasonably well indoors, although not as well as regular high-sensitivity GPS units. The Motorola i580 had the lowest positional accuracy with an RMSE value of 21·64 m. When looking at the results for the 95th percentile, the accuracy of both mobile phones was much less than for static outdoor conditions, but still within the E911 performance requirements.
4.4. Observed versus estimated accuracy
Each position fix on the mobile phone is accompanied by an estimate of the horizontal and vertical accuracy of the position fix. No information is provided by the network carriers or mobile phone manufacturers on how these estimates are derived, but in a regular GPS unit such reliability metrics are commonly derived from the values for Dilution of Precision (DOP), which in turn is based on the number and geometry of satellites whose signals are used in the position estimate.
Figure 4 shows two example scatter plots for the horizontal position: Figure 4a for the Motorola phone and 4b for the Sanyo phone. In the case of the Motorola phone one major outlier (observed error: 7·9 m – estimated accuracy: 120·1 m) is not shown in the scatter plot. For both phones the correlation between the observed error and estimate of horizontal accuracy is very weak and not statistically significant. The scatter plots for the other tests (not shown) revealed very similar patterns. This is disappointing from the perspective of developers of LBS applications. Estimates of horizontal accuracy could be presented in some meaningful manner to an end user, or the estimates could be used to filter out unreliable points during real-time processing of the GPS data. Since other measures of GPS performance (e.g. DOP, number of satellites, Signal-to-Noise Ratio SNR)) are not typically available from the JSR179 Location API, the estimates of positional accuracy are the only measure that LBS applications on mobile phones can utilize. The results in Figure 4 do not support the use of these estimates since there is virtually no correlation to the actual positional error.
Reliability of accuracy estimates for A-GPS data obtained from cell phone on 19 July 2007. Left: Motorola; Right: Sanyo.
The results in Figure 4 allow for an analysis of whether the specifications of the JSR179 Location API are met on the two phones evaluated in these tests. These specifications state that “the true position should be within a circle defined by the given estimated error uncertainty radius at 68% confidence level”. In terms of Figure 4 this means that the observations in the scatter plot should fall above the 1:1 diagonal at least 68% of the time. In the case of Figure 4a this number is 18·6% while in the case of Figure 4b this number is 55·1%. This indicates that the specifications for the JSR179 Location API are not being met on the Motorola i580 and Sanyo 7050 cell phones. In the case of the Motorola phone in Figure 4a the horizontal accuracy was estimated at 2·1 m for 79·1% of all observations during this test while the observed error was much greater than that for most of theses observations. The horizontal accuracy estimate provided by the Motorola phone is biased in the sense that on average it underestimates the observed error while no such bias is observed for the Sanyo phone.
While the lack of a strong relationship between observed and estimated accuracy is bad news for developers of LBS applications, the good news is that under good conditions the horizontal error of valid GPS position fixes on mobile phones rarely exceeds 25 metres even if the estimates for horizontal accuracy exceed this value. It should also be noted that the method of calculation for the estimated accuracy value may differ between manufacturers, or even between different model phones from the same manufacturer depending on the hardware and software implemented in a cell phone, and therefore the results reported in this paper are specific to the Motorola i580 and Sanyo 7050 cell phones and cannot be generalized to other devices which implement the JSR179 Location API.
5. CONCLUSIONS
The performance of GPS-enabled mobile cell phones was determined under varying conditions and compared with regular recreational grade GPS units. Both availability of a reliable position fix as well as positional accuracy were determined.
The availability of valid GPS position fixes on mobile phones during both static and dynamic outdoors tests and static indoors tests was consistently close to 100%. Consistent position fixes are shown to be possible using a high sensitivity GPS-enabled mobile phone, even indoors, which is a key characteristic for meeting E911 requirements as well as for more general LBS applications which must always have some level of position information to operate.
In static outdoor tests the median horizontal error of position fixes from mobile phones varied between 5·0 and 8·5 metres, and the RMSE varied between 6·0 and 12·5 metres. The maximum error during all static outdoor tests never exceeded 30 metres. No significant differences could be observed between the two types of mobile phones (Sanyo 7050 and Motorola i580). These numerical results correspond very closely to results obtained by Zandbergen (Reference Zandbergen2009) for Apple's 3G iPhone, suggesting that the effects of the specific manufacturer and network carrier on the performance of A-GPS are limited. In static indoor tests the horizontal error of position fixes from mobile phones increased somewhat, although limited sample size prevented a more rigorous statistical comparison.
The horizontal error of position fixes from mobile phones was substantially greater compared to those from regular autonomous GPS units under all conditions tested. However, median and RMSE values were typically no more than two to three times higher and therefore still resulted in acceptable positional accuracy to meet E911 performance requirements as well as the needs of most LBS applications. In fact, whenever a valid GPS position fix was obtained, the maximum positional error never exceeded 100 metres, even when considering the indoor tests.
The estimated positional accuracy of GPS position fixes on mobile phones correlated very poorly with the observed positional error. This greatly reduces the value of these estimates for developers of LBS applications and limits the options for developers in terms of filtering poor position fixes while doing real-time GPS data processing. Without access to more traditional metrics of GPS performance (e.g. DOP, number of satellites, SNR values) this restricts the development of LBS applications which require a robust measure of the reliability of GPS position fixes. It should be noted that the methods of determining the estimated accuracy for a GPS fix may differ between different manufacturers or even between different phone models by the same manufacturer, and therefore other mobile phones which feature the JSR179 Location API may produce more reliable estimated accuracy values.
This is believed to be one of a handful of empirical validations of the performance of position fixes obtained using GPS-enabled cell phones. Results demonstrate relatively good positional accuracy and good availability of reliable position fixes, even when indoors. Positional accuracy does decrease, as expected, when the phone is located indoors. Future research in this area should consider the performance under varying field conditions, including urban canyons and different types of indoor environments. Future research should also examine newer model phones from Sanyo and Motorola as well as phones from additional manufacturers in order to determine if any unexpected hardware or software issues in the first generation of high-sensitivity GPS-enabled cell phones affect positional accuracy and also to determine if positional accuracy is consistent among different model phones from a variety of manufacturers. Future research can also examine the positional accuracy of GPS-enabled mobile phones when GPS data is stored locally on the phone instead of being immediately transferred to the server via wireless communication in order to reduce the chances of data being lost in transit. The reliability of estimated accuracy values for other make and model cell phones should also be evaluated to determine if the lack of correlation between observed and estimated accuracy is present on other GPS-enabled mobile phones.
6. LIMITATIONS
A number of limitations can be identified with respect to the current study. First, for the mobile phone tests, several communication and software issues occurred during testing which prevented the transfer of some location data from the mobile phone to a server for storage during testing. Since there is not believed to be a bias towards one type of location data being lost during transit more than others (e.g. location of cell phone tower or GPS), these issues effectively reduced the available sample size for several of the mobile phone tests.
Second, the static indoor and dynamic outdoor tests rely on digital ortho-imagery to derive the reference locations used in the accuracy determination. While this imagery represents the current state-of-the-art in digital aerial photography, there is some inherent error in using them to determine reference conditions.
Third, the number of tests employed in this study is relatively small, which prevented certain types of analysis. For example, no statistical differences were found in the positional accuracy of the positions from the two different types of mobile phones. A larger sample may still establish a statistically significant difference.
Finally, only a limited set of indoor conditions were evaluated, reducing the general reliability of the results for indoor positioning. Further research into the performance of A-GPS under indoor conditions would be very relevant given the FCC requirements for the performance of mobile phones.
