MARS: An Augmented Reality-Based Marine Chart Display System

Pratomo, D. G.,^1* Khomsin ¹ and Aditya, N.²

¹Geomatics Engineering Department, ITS Surabaya, Indonesia

²Geosolution Pratama Nusantara Ltd, Indonesia

*Corresponding Author

Abstract

The visualization system of nautical charts enhances the safety navigation on sea, particularly in critical areas. Nowadays nautical chart visualization technology considers the benefit of inventive information technology to produce three-dimensional and real-time models which can be easily used. Therefore, this study attempted to develop an Android-based application with Augmented Reality (AR) approach. A digitation of the conventional nautical chart was performed to build a digital nautical chart. This process implemented regulation from United States (US) Chart No. 1 and International Association of Marine Aids to Navigation and Lighthouse Authorities (IALA) Maritime Buoyage System. There are two substantial results of this research: Marine Augmented Reality System (MARS) to display a nautical chart in a three-dimensional form, and Marine Augmented Reality System GPS (MARSGPS) which focuses on integrating the model with its vicinity environment. This brings users to better acknowledge the situation of their environment. The usability test of these products shows that the overall feasibility value is 83.71%. The coordinate accuracy of MARSGPS application has a Root Mean Square Error (RMSE) value of 4.30m, which can be improved through a ground-truth measurement. Nevertheless, this approach provides an essential reference for future research on a three-dimensional nautical chart visualization.

Keywords: Android-based Application, Augmented Reality, Nautical Chart Visualization, Safety of Navigation

1. Introduction

Nautical charts are unique purpose maps specifically designed to meet the requirements of marine navigation, showing, amongst other things, depths, nature of the seabed, elevations, configuration and characteristics of the coast, dangers, routes, maritime limits, and aids to navigation. Therefore, it represents a graphical representation of relevant information for seafarers in terms of planning and executing safe navigation [1]. One crucial factor capable of describing the numerous substantial aspects in the actual condition of a nautical chart is the accuracy of each aspect's coordinates. Moreover, assisting the mariners in discriminating one sea feature from another becomes another point to provide a convenient nautical chart. Those considerations are essential when the ships reach a shallow water area. The definition of shallow water must depend on the characteristic of the local wave. However, according to the Publication of IHO S-44 sixth edition [2], the critical areas of shallow water include harbors, berthing areas, and areas of fairways and channels. A particular harbor that is specialized for the trading area, port, is described further as a location on a coast or shore where ships can dock and transfer people or cargo to and from land [3] and [4].

Generally, it is described as a location in a town or city with a harbor or access to navigable water where ships load or unload [5]. In practice, these various facilities are displayed on a digital nautical map that, in real-time, shows the direction and location through the ship's monitor, Electronic Chart Display Information System (ECDIS). Following IHO Publication S-57 [6], this device displays two-dimensional water areas, so it still requires intuition for seafarers to understand the location of the ship's presence and ensure obstacles in shipping safety. The location of this research, the Madura Strait, is a semi-enclosed sea that separates the island of Java from the island of Madura [7].

Geographical conditions indicate the strait as a vital water area with various loading and unloading activities to meet market demand in eastern Java and eastern Indonesia.

According to the function of strategic water areas, there are various port terminals, such as Lamong Bay Terminal, Gresik Petrokimia Terminal, and the Surabaya Tanjung Perak Port area. In addition, there is also a reasonably busy sea transportation route known as the West Surabaya Shipping Line [8] and [9]. The density of ship traffic requires good navigational aids to reduce the risk of accidents around the waters. Based on Law no. 17 of 2008 [10] concerning shipping, navigation is the process of directing the motion of a ship from one point to another safely and smoothly and avoiding hazards and obstacles to shipping. Meanwhile, according to Government Regulation no. 5 of 2010 concerning navigation, shipping navigational aids are equipment or systems outside the ship that are designed and operated to improve the safety and efficiency of navigating ships and ship traffic. The digital navigation devices can be improved by displaying shipping lanes in 3D form in accordance with the 2018 IHO S-100 Publication [11]. In addition, various technologies in the industrial era 4.0 support the improvement of the quality of digital marine maps such as artificial intelligence, geo-big data, cyber- security, and autonomous ships. Augmented Reality (AR) itself is a technological development that can display an object generated from a virtual computer to a real-world environment in real-time. By combining this technology with location information using Global Positioning System (GPS), a Three-Dimensional (3D) and real-time visualization model will be obtained. In the study of Boghosian, et al [5], AR technology has been developed in the field of Geographical Information System (GIS). Meanwhile, in the field of shipping safety, there is an integration system between Augmented Reality (AR) and ship GPS called the Automated Identification System (AIS) [12].

Therefore, in this study, an Android-based application will be built by utilizing AR technology to support shipping navigation safety, especially in the Madura Strait area, East Java, Indonesia. The AR system developed using Unity3D software will later display 3D objects following the provisions of the international buoy system. It will then be tested using an assessment research method to users through questionnaires for the feasibility level based on the usability test results under the feasibility category table.

2. Material and Methods

The basis of this research uses data in the form of a marine map, namely the National Sea Map Sheet Number 84 of 2016 [13], scale 1:12500, which is located in the waters of the Madura Strait, more precisely around Lamong Bay with an area of about 134 Km². The coordinate limits of the research area are 7°8′17.42" - 7°13′38.64" S and 112°38′47.81" - 112°46′11.54"E as shown in Figure 1.

Figure 1: Lamong Bay as research area

Figure 2: Sea features digitalization process

In addition, the 2018 National Bathymetry Map (BATNAS_110E-115E_10S-05S_MSL-v1.5) was used to support the representation of the subsurface waters. The following process is building a 3D model using SketchUp software considering the United States (US) Chart No. 1 and International Association of Marine Aids to Navigation and Lighthouse Authorities (IALA) Maritime Buoyage System to represent the chart features differently from one another. In addition, national bathymetry maps also require transformation or modeling into 3D form to support the overall 3D modeling. This digitization process uses the Delaunay triangulation principle [12] and [14].

The initial step is to digitalize the nautical chart features that support safety of navigation, such as buoys, sea frame beacons, berth areas, and fishing areas (Figure 2). The following process is building a 3D model which considers the US Chart No. 1 and IALA Maritime Buoyage System to represent the chart features differently from one to another. In addition, national bathymetry maps also require transformation or modeling into 3D form to support the overall 3D modeling.

Some initial settings are executed to support the 3D modeling process and the creation of Augmented Reality AR, which consist of 1) Compiling the Unified Modeling Language (UML) based on IHO S-100, 2) creating an AR Marker in the form of a Quick Response (QR) Code, 3) accessing Vuforia, an AR service, to get the detection value from the QR Code, license manager, and database manager, and 4) setting the device Unity3D software includes inserting a license manager and database manager, as well as changing the platform to Android. Then, the 3D modeling results of each of these features need to be arranged by considering the scale of the model and referring to the 8 Golden Rules in User Interface/User Experience (UI/UX) design [15] and [16]. The process of developing the UI/UX design produces a 3D model that is adapted to a smartphone display. The results are then exported into *.apk file format with the name Marine Augmented Reality System (MARS) so that it can be installed and operated using an Android-based smartphone device. To assess the function, application effectiveness, and user satisfaction, a usability test with a weighted value is a necessary. The weighted value is classified into five different number based on Likert Scale, as shown in Table 1 [17]. Calculating the feasibility application in percent (%) is based on the Nielsen equation [18] with the classification of the results of the feasibility value based on Likert [17]. Data was collected in this usability test according to Table 1. Suppose the results show a value of less than 60% or less than the appropriate category.

Table 1: Weighted value

Table 2: Feasibility classification

In that case, it will return to the 3D modeling process by considering user input through the questionnaire results. So that the results will be obtained in the form of the appropriate android based MARS.apk and MARSGPS.apk applications. Determining the percentage of aspects and usability’s value are described in Equation 1 and Equation 2 as follows:

Equation 1

Equation 2

where ais the learnability aspect, b is the flexibility aspect, c is the effectiveness aspect, d is the Attitude aspect, and x_i is the amount of the I successful respondents, x_i =[0,1, …, n]. Table 2 shows the usability level classification of the AR application. Then the coordinate accuracy of the models and the real-time location need to be calculated using Root Mean Squared Error (RMSE), which can be seen in the Equation 3 below: [19] and [20]:

Equation 3

where x_dis x coordinate in nautical chart, y_d is y coordinate in nautical chart, x_c is x coordinate from Global Navigation Satelite System (GNSS) measurement, y_c is y coordinate from GNSS measurement, and n is number of samples. In this research, the validation data was collected from coordinates in a public domain imagery data. It might affect the coordinate validation process then. Base on the digitation process as shown in Figure 2 there are two types of features in the area of interest which are point and area. The number of each object which are digitized in the Surabaya Tanjung Perak Port area can be seen at Table 3.

3. Results and Discussion

In this section, the results of each process will be explained. Various discoveries in every process point will be discussed, including digitalizing features, three-dimensional (3D) modeling, the application, and the usability test for users.

3.1 Digitized Features

The initial step utilized as a base of the next step is the digitized features. Information on each feature on the marine map is shown in Table 3 and Table 4. The feature numbers result from the digitization process of the National Nautical Chart number 84 of 2016, particularly in the research area. All features are located in the Surabaya West Shipping Channel and are spread around ports along the channel, such as Lamong Bay Terminal.

Table 3: Digitation results summary

Table4: Feature information in the nautical chart

3.2 Augmented Reality (AR)

Three-dimensional modeling is a crucial stage in building AR because it will determine the level of clarity of the information presented in nautical charts.Table 5 provides an overview of some features in the 3D modeling process, namely, buoy categories and lighthouses. This modeling process considers the provisions of the IALA Maritime Buoyage System NP735 edition 5 of 1994 concerning the Combined Cardinal and Lateral System and the U.S. Chart Number 1. In addition, 3D modeling also references Tritschler's research [21] to complete the dimensions of each feature. This case is determined due to the National Nautical Chart's limitations in the feature’s height dimension.

3.3 Augmented Reality

Building Augmented Reality (AR) begins with designing the User Interface (UI). The design should

include essential parts such as Main Menu, Feature Models, and the About tab. Furthermore, it needs a UX design prepared using Unity3D software to complete the application. The process designs specific sections, for instance, AR Scene and Simulation, or the UI main parts. Moreover, the based-location element of the application should be designed with GNSS assistance, for example, Location Info, Latitude, Longitude, and many more.

Table 5: Three-Dimensional modelling feature results

The results of the AR technology implementation are two mobile-android-based applications, namely MARS and MARSGPS. The MARS application uses a QR Code to start the visualization, registered in Vuforia software earlier (Figure 3). Using image tracking, users can interact with the 3D model by zooming in or out and rotating clockwise or counterclockwise in real-time by adjusting the scale in the bottom right corner. Also, the users can return to the main menu with the back button in the upper left corner. Figure 4 Visualizes the 3D model from the image-based MARS application by scanning the QR Code.

In addition, some sea features happen as coordinate verification between the actual condition and the resulting model. Actual coordinates in this research are not measured but observed through remote sensing secondary data from a public domain imagery. This raises awareness that it could be one of the weaknesses. Furthermore, this condition recommends that future research using actual, or ground truth measurements decrease the deviation in centimeters. Table 6 shows the difference between the actual and the model coordinate on some pointed navigation features, namely, buoy categories and a lighthouse, as the several 3D models on Table 5.

For the MARSGPS application, the 3D model will be displayed on the smartphone screen and the surrounding environment in real time. Therefore, the actual model location could also be presented through the application. The real-time segment of the model as well as implemented when rotating the device. The visibility range’s result shows all the features within the nautical chart's actual range. Figure 5 shows that the MARSGPS application is carried out in the Department of Geomatics Engineering of ITS's field according to the coordinates entered in the 3D model display. The users could customize the model’s placement regarding the area.

The displayed 3D objects will shift following the updated position because the MARSGPS application is integrated with the smartphone's GNSS. There are some limitations due to the smartphone’s GNSS precision [22] and [23]. Thus, the precision of location determination can range from 2 to 5 meters in good conditions and weather, and in certain conditions, it possibly ranges from 10 to 20 meters [24]. Table 7 below describes the result’s differences between MARS and MARSGPS according to several categories.

Figure 3: QR code according to Vuforia software

Figure 4: MARS visualization result after scanning the QR code

Table 6: Comparison between the actual and model coordinates

Table 7: Comparison between the result applications

Figure 5: (a) MARSGPS Application presenting the location information and (b)three-dimensional modelling

Table 8: Learnability aspects

Table 9: Flexibility aspects