1. INTRODUCTION
The number of people killed in pleasure craft accidents in Norway has been relatively stable for the last ten years ranging between 38 and 24. In 2015 the number was 37 and in 2016, 27 (Berntsen, Reference Berntsen2017). The reduction in deaths by 2016 could be the result of the mandatory requirement to use a life jacket in small boats of less than 8 metres from May 2015. The majority of these accidents happened in confined coastal waters among the approximately 50,000 islands along the Norwegian coast. Around 20% of the accidents with a deadly outcome were caused by groundings (Berntsen, Reference Berntsen2017). The presented statistics on fatal accidents hide a much larger number of accidents where nobody was killed but which still involved hull damage. The Norwegian Society for Sea Rescue published statistics in August 2017 on the number of groundings in the county of Ostfold (south-east of Oslo). 223 leisure boats had to be towed off rocks in Norway by the Sea Rescue Society between January and August 2017 (Norwegian Society for Sea Rescue, 2017). A survey by a Norwegian insurance company revealed that 40% of leisure mariners in a county in mid-Norway could not read a nautical chart (Hitra-Froya, 2016).
1.1. Background
In 2008 the main author participated in a conference about safety of leisure craft in Oslo. During a session, a representative of a Norwegian pleasure craft organisation said something along the lines of “what boat people want is not complicated chart machines, but a simple black box in the pocket which sounds an alarm 30 seconds before they run aground”. There was much laughter, and nobody really took it seriously. However, we remembered this remark as we continued working on ways of making leisure navigation more robust to training deficiencies, inexperience and time stress. Much of the work concentrated on ‘cognitive off-loading’.
1.2. Cognitive off-loading
We humans view and act in our awake life from the egocentric perspective of our eyes. Maps, however, are depicted from an exocentric birds-eye perspective. Navigation is much about translating own position and intended course from the map perspective to the real world. The projection change that needs to be done on the fly in the head is called ‘mental rotation’ and is both error-prone and cognitively demanding (Porathe, Reference Porathe2006). One way of facilitating navigation would be to let the map system conduct this change in projection, presenting the map in an immediately useful egocentric 3D-view (see Figure 1).
A concept illustration of an egocentric map view is shown on the right screen as compared with the traditional exocentric map view of the same situation on the left screen. (Illustration Porathe, Reference Porathe2006)
Porathe (Reference Porathe2006) showed that the way-showing task of navigation was more effective (allowing higher speed of decision-making and producing fewer errors) using egocentric view maps. At the same time, the traditional exocentric perspective was better for route planning and other tasks requiring an overview.
1.3. Three-Dimensional (3D) and augmented reality maps
The original suggestion by Porathe (Reference Porathe2006) was to add new information to the map database (terrain elevation, conspicuous buildings, etc) and then allow the map system to seamlessly change between egocentric and orthographic exocentric views according to the users' needs. However, another way of achieving a simplified egocentric view would be to add chart information in an Augmented Reality (AR) layer on top of the camera image in an ordinary smartphone available to many people today. AR or Mixed Reality (MR) are when computer generated imagery, geo-referenced or not, are placed on top of a real world view in, for example, heads-up or head-mounted displays, but also in cameras and touch pads.
2. METHOD
In October 2016 the SikkerKurs-project started with a budget of 1·5 million Norwegian crowns financed by the Norwegian Coastal Administration (Kystverket) and Geomatics NORWAY AS. The purpose was to develop an AR application for ordinary smartphones to increase safety and possibly decrease the number of groundings by leisure boats in Norwegian waters. The project was to be a proof-of-concept demonstrator, coordinated by Geomatics Norway AS. The designer was the first named author, working at the Norwegian University of Science and Technology (NTNU) in Trondheim and technical implementation was conducted by Combitech AB in Linkoping, Sweden. Other partners in the project were the Norwegian Hydrographic Office (Kartverket) and the Norwegian Maritime Authority (Sjofartsdirektoratet). It was decided that the project would use the Human-Centred Design process (HCD) in ISO 9241 (ISO, 2015) and IMO's guideline on HCD (IMO, 2015).
2.1. The Human-Centred Design process (HCD)
The point of HCD is to ensure user-driven development and a good usability by including the end users early in the process and keeping them involved during the whole design process. This is done in an iterative process with four steps:
(1) Understand the context of use by field studies and interviews with the users.
(2) Specify the user and organisational requirements.
(3) Produce a design solution; this will be the prototype.
(4) Evaluate the design against requirements. Here the prototype is tested on the end users.
The findings are then brought into a new iteration of the design process resulting in a new, improved prototype. The process is then iterated until the application meets the requirements.
2.2. Test area
Before a user group could be recruited, a location had to be decided on. One way would be to look for an area with a large amount of leisure boat traffic. However, the availability of detailed bathymetry and chart data was a precondition. The Hydrographic Office offered an area in Sore Sunnmore, a district south of Aalesund on the Norwegian west coast which had been declassified and could be used. A central municipality in this area was Ulsteinvik, which was to become the centre of the project.
2.3. Recruiting a user group
We needed to come in contact with local leisure boat mariners to recruit the user group. A letter was sent out to 30 pleasure craft clubs in the district informing about the project and asking about participation in development and testing of the application. The concept image in Figure 2 was included to illustrate the idea.
Concept image for the project.
Only six leisure boaters responded, all male, all relatively experienced and with an age of 60+. Although we had hoped for a larger user group with mixed ages, gender and experience, this was the user group we had to start with.
2.4. Understand the context of use and user requirements
A first focus group meeting was held in Ulsteinvik in January 2017. The users were interviewed about their experience with leisure craft navigation and the proposed concept of using a smartphone as a means of preventing groundings was discussed. The group concluded that the idea was very interesting and that there was a need for a safety device alarming if a boat was approaching unsafe depths. The group agreed on a prioritised list with different possible features.
2.4.1. Alarm
The phone should sound an alarm a configurable time before the boat went aground. The application should be automatically started in the background when a boater steps into his boat, so that he or she does not forget to start the application. The time should be short so that the number of false alarms in narrow archipelagos would not be annoying and thus making boaters turn off the alarm (which is often the case with the look-ahead-sector in professional shipping). The default setting was agreed as 30 seconds, and the procedure of the boater should be to immediately stop the boat on alarm. The alarm should be silenced by picking up the phone and clicking the warning icon shown. The alarm should also be silenced by slowing down to a configurable maximum speed (default 3 knots) to allow boats to make landfalls or approach a jetty without getting an alarm.
2.4.2. NoGo areas
When the phone is picked up and the alarm silenced, the screen should show “NoGo areas” in red overlayered on the camera image. These NoGo areas are the polygon inside a configurable depth contour. The default was the 3 metre contour, but ideally, any depth should be able to be picked based on the current draft of the boat, plus a safety margin. Ideally, the depth alarm should also compensate for the current tidal situation based on tide tables or real-time tide gauges. The user should be able to see these NoGo areas all around by pointing the smartphone camera.
2.4.3. Landmark names
Conspicuous landmarks around the boat should be named by overlaying text on the camera image. Examples of such conspicuous landmarks could be names of islands, shoals, beacons and mountaintops (the area is very mountainous). Much time on board a small craft is spent trying to find buoys and beacons. An overlaid pointer should show their position to aid visual search. To avoid cluttering, the names and pointers could be toggled on and off by tilting the camera (slightly up turns text on and vice versa).
2.4.4. Air draught
An alarm similar to the grounding alarm could be configured for sailing boats with a mast height that is higher than the span of bridges and power lines.
2.4.5. Fairways and planned routes
Official fairways should be shown as a “carpet” rolled out on the water in the camera image. Also, individual routes planned in a chart program and imported into the phone might be shown. This feature must be able to be turned on and off to avoid cluttering. This requirement was later dropped for the tested prototype due to time constraints.
2.5. Technical prototype development
After the meeting with the user group, discussions started about the technical implementation and what could be achievable within the time and budget available. Of the five prioritised solutions suggested by the user group the first four were selected for development.
2.5.1. The Android platform
We decided to make the test implementation on the Android platform because Combitech had earlier experience of this platform, had available equipment and the relative ease with which test implementations could be distributed without being passed by the App Store (for Apple's iOS), thus giving us a quicker development cycle.
2.5.2. NoGo areas and alarm execution
Part of an Electronic Navigational Chart (ENC) was imported into a database in the phone's memory. From the ENC only the polygons making up the area with a water depth of less than 3 metres at chart datum were kept. These polygons made up the “NoGo area” that was used to alarm the navigator for grounding. Ideally, we would have NoGo polygons for every centimetre, which could be turned on and off depending on the set draught of the boat and the tidal situation. However, this would require large memory storage or a constant on-line connection, so we decided to have just one NoGo depth of 3 metres for the testFootnote 1. The Norwegian Hydrographic Office delivered the necessary depth contour on a high-resolution horizontal grid of 1 metre dimensions. The internal map would consist of this area of water depths between 3 metres and the beach line.
The timed alarm function was implemented using a vector extending from the present position in the direction of the current course. The length of the vector was dependent on the speed and the alarm time set. In the default setting the alarm was set to be triggered 30 seconds before the boat “grounded” (passed into the 3 metres NoGo area polygon). At 10 knots the length of the vector would be (10 knots * (1,852 metres/3,600 seconds) * 30 seconds) = 154 metres. The length and direction of the course-speed vector was calculated from recent Global Navigation Satellite System (GNSS) positions. The precision was dependent on the position rate the phone could muster, which in general was one position per second or less. The alarm would be triggered when a course-speed vector intersected with a NoGo area polygon.
The air draught alarm was treated the same way using the same course speed vector intersecting a safety rectangle extending 15 metres on both sides of bridges and power lines. The set mast height would then be compared against the maximum air draught allowed as stated as an attribute to the safety rectangle. In the test area there was only one power line and no bridges.
2.5.3. The Augmented Reality (AR) layer
The NoGo area polygon map was to be shown on top of the camera image at the correct position. The polygons should apparently be “floating” on the surface of the water. In order to do this, the map had to be georeferenced and projected using a virtual camera positioned in virtual space as the real camera was in the real space. This projection is a standard Virtual Reality (VR) operation conducted in real-time taking the virtual camera's height over the water (pre-set to 2 metres), direction (from the phone's compass) and field of view (pre-set to match the device's camera) as in-parameters. In order to keep computation time low and not to clutter the display, the maximum visibility of the NoGo areas was set initially to 2,000 metres by the means of a clipping plane.
The course-speed vector was also made visible and projected into the camera view; white when not in alarm mode but changing colour to red when an intersection had taken place and the alarm was triggered. It was then red as long as it was intersecting with the NoGo polygons, thus visualising the alarm state, and remained red when the aural alarm was silenced. The initial intersection point was shown by an arrow.
The stability and precision of the GNSS positions and the compass heading from the internal phone sensors was an area of concern. The course-speed vector triggering the alarm was created by extrapolating the present course and speed into the future. Low-pass filters were applied to these values to avoid large jumps due to unstable GNSS fixes. This was done to reduce the risk of false collision alarms. The point of view in the polygon map was also dependent on the GNSS-based present position, but the direction of the camera (which was independent from the course-speed vector of the boat) was reliant on a compass direction from the phone's internal magnetic compass. We had little experience of the precision of these two sensors, which might also be dependent on local conditions in the area for the test. However, to anticipate possible problems with the compass, we made it possible to shut down this sensor and use the course speed vector as direction for the virtual camera in the augmented reality layer, then assuming that the camera was fixed in a forward-looking manner (for example, on the windscreen).
The only text-based information we considered we had time and resources to implement was the pointer for navigational marks. The position of all buoys and marks in the test area was collected in a list. We did not succeed in populating the list with all the marker names in time, so the markers in the test's prototype mostly showed “POI” for point of interest.
3. RESULTS
The first iteration of the prototype was tested during a technical test in Ulsteinvik with two people from the user group on 8 May 2016. The full user test was conducted a month later on 14 June.
3.1. Technical test
For the technical test in May a relatively complex, 5·8 nautical miles long track was drawn in an ENC (see Figure 3). This track could be negotiated in a little more than an hour at a moderate speed of 5 knots (not to take any risks should the prototype prove unreliable).
The test track west in Ulsteinvik in western Norway
For the test, we used a 7 metre leisure boat owned by one of the users. He had also very good local knowledge, which would be a safety barrier against unintentional grounding should the prototype fail. The boat was equipped with a fixed chart plotter which was used as a reference system (see Figure 4).
The test application on the smartphone (Lenovo Phab 2 Pro) during the pilot test. To the right is the boat's reference fixed chart plotter.
The prototype software was tested on two phones, one Samsung Galaxy S7 and one Lenovo Phab 2 Pro. We found no differences in behaviour between the two phones. Some problems with the fluctuating AR layer are described below.
3.2. User test
The final user test was held in Ulsteinvik on 14 June 2017. The same test track as in May was used and all six of the original users were present on the 15 metre M/S Legona used during these tests. The boat made the passage at about 5 knots in just over an hour and the prototype was tested on the two phone types mentioned above. Below are the results of this user test.
3.2.1. Alarm execution
The function to automatically turn on the application when leaving port was not developed for this first prototype. The application was manually turned on when the test run commenced. When the course-speed vector intersected the NoGo area the alarm was triggered, both while the phone was “sleeping” in the pocket, or (as in Figures 5, 6 and 7) when the phone was used to actively monitor the water ahead of the boat. By touching the stop sign, the alarm is acknowledged and silenced and the stop sign disappeared. However, the vector remained red as long as it was intersecting a NoGo area. This feature worked perfectly as designed and the comments from all the users were very positive.
Screen dump from the Galaxy test phone. The projected NoGo areas are in red. The 30 second course-speed vector is in white just before the alarm is triggered. The pointers show three points of interest (two of which are hidden behind the island).
Screen dump from the Galaxy test phone. The grounding alarm has been triggered with both an aural and a visual alarm.
A very narrow passage on the test track. The distance between the 1-metre shoal and the small skerry is 33 metres (in the chart, left). Right, the app view entering the narrows (northbound).
3.2.2. NoGo areas
The augmented reality layer was projected over the camera image based on a virtual camera positioned by latitude and longitude from the phone's GNSS sensor and the virtual camera's direction was based on input from the phone's internal magnetic compass. Both these sensors had fluctuations as opposed to the camera image which of course moved only when the phone moved. This resulted in smaller or larger fluctuations of the AR layer over the camera image. The AR layer with the NoGo areas, course-speed vector and POI pointers would float or jump in the image, mostly in the horizontal plane. These fluctuations would be more or less extreme depending on factors such as if the camera was being panned, and/or there were magnetic disturbances in the boat or in the area. The sensitivity to magnetic disturbances is illustrated by this example: One phone tested had a leather cover that could be closed over the screen with a magnetic lock. This lock jammed the compass causing the AR layer to become unreliable.
The horizontally fluctuating AR layer was the one disappointment in an otherwise successful test. Ideally, the layer, with its added information should be steady and “glued” to the camera image. In the test prototype it jumped or sailed some 5–10° to either side of its intended position. However, the user group judged it to be within reasonable limits. This was because the inside of the NoGo areas was visually easy to pair together with the island's beach line, making the fluctuations “some kind of visual expression of uncertainty” (user comment). In Figure 8 an offset to the right and slightly up can be seen. The beach line of the island and the front beach line in the inner hole of the red NoGo areas match. Note also that the NoGo areas behind the island are visible and they should not be. Theoretically they could be clipped using an invisible 3D terrain model in some future version of the app. This 3D terrain model could then be shown during darkness and fog when the camera was out of play. However, the most important thing was that the triggering of the grounding alarm function was not affected by the fluctuations due to the magnetic compass. The alarm computation was done entirely in the map layer using the relatively more stable GNSS position.
The picture shows the offset of the AR layer with the red NoGo areas. There is a vertical offset and a horizontal and fluctuating offset due to noise in the phone's internal compass. However, users judged it acceptable during the tests.
3.2.3. Points Of Interest (POI)
The pointers to named points of interests (for example, light houses, buoys and other marks) are potentially beneficial as a second source of information to cross check the visual integrity of the system. However, this feature was not tested as we did not get access to names of the markers in the area (which were not present on the chart). In the prototype, most marks only carried an anonymous “POI” label. This feature will be investigated further.
3.2.4. Survey
After the test voyage and a short debriefing, the six users answered some questions in a small survey. The first question was whether they thought that the tested prototype could have any favourable effect on boat navigation. On a scale from 0 − 100, where 0 was “no favourable effect” and 100 “large favourable effect” they were asked to indicate their answer with a cross. The mean result of all six users was 83. Close to “large favourable effect”.
The second question dealt with the usability of the prototype application. On the same type of scale from 0-100, where 0 was “simple to use” and 100 was “difficult to use”, they were asked to mark their answer with a cross. The mean result from the six users was 13, clearly on the “simple to use” side.
They were also asked to comment on the prototype and asked if they missed any functions. Three answered “no”; one gave no answer; and the remaining two made these comments: “The matching between the AR layer and the camera image could be better”, “Automatic Identification System (AIS) data could be added”, “Some adjustments and it will be fine”, “Get it out as soon as you can, new versions can come later.”
During a concurrent television interview, one of the users commented on the alarm function: “I am often out sailing in my boat and when tacking we often want to use the water between the islands as much as possible, and then often go close to land. If we could get an alarm by a buzzer in the pocket instead of having to constantly look on our navigator screen, that would be great.”
The users were also asked about the likelihood that they would use such an app if it was (1) free, (2) costed the equivalent of £2·50 and (3) £25. On the same scale from 0-100 they answered that the likelihood was (1) 95, (2) 94 and (3) 82. Which translates to a high willingness to pay a reasonable amount for such a safety app.
4. DISCUSSION
The intention of this project has not been to develop a system to replace traditional navigation but to create a “last line of defence” against accidents. However, it will be difficult to prevent a few boaters from using it as a sole means of navigation. The question is then: If we develop a “simple, stupid” application, which facilitates boating for leisure mariners without navigational training – do we then lure new “unfit” groups of people out on the sea, which in the end might lead to more accidents? And, do we contribute to the de-skilling of leisure mariners?
Let us make a parallel with professional navigation. Traditionally ship's positions were acquired by measuring the angles to the sun or terrestrial landmarks. After some calculations you obtained a “historical” position, where the ship recently was. This position was then manually plotted onto the paper chart. There were abundant opportunities of making errors during the measurement, the calculations or during the plotting, let alone that overcast days or bad visibility sometimes made measuring impossible.
When the radio-based Decca and Loran systems and later the Global Positioning System (GPS) came, the measuring process was automated and only the manual plotting into the chart remained until Electronic Chart Display and Information System (ECDIS), allowed the officer to have the ship's position automatically plotted on the chart in real-time. In 1989 the International Maritime Organization (IMO) issued the first provisional performance standards for ECDIS (IMO, 1989) and in 1995 the US Coast Guard presented an early human factors study (Smith et al., Reference Smith, Akerstrom-Hoffman, Pizzariello, Siegel, Schreiber and Gonin1995). This concluded that “ECDIS had the potential to improve upon the safety of navigation, compared to conventional procedures,” and that “there was strong evidence that the use of ECDIS increased the accuracy of navigation, [—], and reduced the proportion of time spent on navigation, with a corresponding increase in the proportion of time spent on the higher risk collision avoidance task. In addition, ECDlS was shown to improve geographic ‘situational awareness’ and to reduce navigation ‘errors”’ (Smith et al., Reference Smith, Akerstrom-Hoffman, Pizzariello, Siegel, Schreiber and Gonin1995, P.VIII). Spontaneous comments such as: “Navigation goes away as a task” were made by the participants.
However, this was achieved at the cost of what we call de-skilling. No longer did the mariners need to train their skills in taking sun heights with the sextant or bearings with a pelorus. They became more dependent on the automatic systems. In an article in the Journal of Navigation, Edmund Hadnett (Reference Hadnett2008) from the Port of London Authority, reacted to the de-skilling of navigators with the rise in dependence on modern bridge technology leading to “over-confidence in situation awareness, encouraging individuals to take far greater risks than was previously the case where a good look-out and a safe speed were intrinsic parts of watch-keeping”. Hadnett (Reference Hadnett2008) concluded that “The drive to improve safety at sea by the introduction of electronic navigational equipment to enhance situation awareness and assist the watchkeeper has unwittingly compromised safety standards by reducing the core competences that were demanded of previous generations and engendering the undesirable human trait to select the easiest option.”
Furthermore, de-skilling continues, now the ECDIS itself has become complicated. In the foreword of the UK Maritime Accident Investigation Board's (MAIB) report after the Ovit grounding in the English Channel 2013, the UK Chief Inspector of Marine Accidents wrote “This is the third grounding investigated by the MAIB where watchkeepers' failure to use an Electronic Chart Display and Information System (ECDIS) properly has been identified as one of the causal factors.” (MAIB, 2014. P.1)
However, although the observations that the de-skilling among professional navigators are undoubtedly true, the safety and reliability of modern shipping keeps improving from year to year. To provide a perspective, it is interesting to note that in the three years 1833–1835, on average 563 ships per year were reported wrecked or lost in the United Kingdom alone (Crosbie, Reference Crosbie2006). Today, in the world fleet of tankers, bulk carriers, containerships and multipurpose ships, which have risen from about 20,000 ships in 2000 to more than 32,000 in 2016, the percentage of the world fleet totally lost per year (ships over 500 Gross Tonnes) declined from 0·37% in the 2000, to 0·11% in 2016 – and this was worldwide (IUMI, 2017). So, although automation has led to de-skilling, it has also lead to safer shipping. The question now is, can the same argument be made for technology in leisure navigation? We would say yes, and argue that a simple, automated tool, warning leisure mariners against grounding, will potentially result in fewer accidents if properly developed in the process of going from prototype to product.
5. CONCLUSION
In this study, we tested a simple, smartphone-based safety application for leisure boaters. Leisure boaters often have a limited knowledge of navigation, according to accident statistics, and the application was designed to be easy to use and understand without prior knowledge. It worked in two ways: (1) In a “turned off ” mode in the pocket the phone would give an alarm 30 seconds before the boat entered into “dangerous waters” (depth less than 3 metres). The boat owner was then expected to immediately stop the boat. (2) Picking up the phone, the owner could look through the application's camera view and see red “NoGo area” polygons overlaid on the camera image. By looking around, he or she could then detect navigable water and continue the voyage.
The application contains a high-resolution map of the 3 metre depth contour extracted from a nautical chart. This map was then projected on the camera image's “egocentric view” of the surroundings, thus bypassing the potentially cumbersome mental rotations a human navigator has to do when comparing a traditional exocentric map with the world around. This would facilitate use by inexperienced boaters.
The application was tested on a small group of six Norwegian, all male, all experienced, leisure craft mariners. The size and configuration of the test group limits the generalisability of the results, but the group had highly positive views of the tested prototype, which encourages us to continue this work.
Future work includes adding some limited features asked for by the user group while still maintaining a simple and easy to use app. The most prominent new feature will be the ability to import a pre-planned route from a nautical chart application (or an official route from the Coastal Administration) and show this route in the AR layer overlaid in the camera image, thus not only showing dangers to navigation but also offer way-showing.
The intention of this experiment was User Experience (UX), to find out if such an egocentric AR application would be beneficial and would potentially be used by leisure mariners in an archipelago setting. Precise technical benchmarking and testing of different smartphone brands' potentials and problems were not undertaken at this stage but will be a task for future development. The intention was not to replace the normal navigation procedure, but to add an extra safety layer.
FINANCIAL SUPPORT
This work was supported by the Norwegian Coastal Administration (Kystverket) and Geomatics NORWAY AS.
ETHICAL STANDARDS
The authors assert that all procedures contributing to this work comply with the ethical standards of the relevant national and institutional committees on human experimentation and with the Helsinki Declaration of 1975, as revised in 2008.
