Vertical Accuracy Assessment of Improvised Global Digital Elevation Models (MERIT, NASADEM, EarthEnv) Using GNSS and Airborne IFSAR DEM

Aziz, M. A. C.,¹ Pa’suya, M. F.,^1* Talib, N.,¹ Din, A. H. Md.,^2,3 Hashim, S.¹ and Ramli, M. Z.⁴

¹Environment and Climate Change Research Group (ECC), College of Built Environment

Universiti Teknologi MARA, Perlis Branch, Arau Campus, 02600 Arau, Perlis, Malaysia

²Geospatial Imaging and Information Research Group (GI2RG), Malaysia

³Geoscience and Digital Earth Centre (INSTeG), Faculty of Built Environment and Surveying

Universiti Teknologi Malaysia, 81310 Johor Bahru, Johor, Malaysia,

⁴Institute of Oceanography & Maritime Studies (INOCEM), Kulliyyah of Science,

International Islamic University Malaysia, 25200, Kuantan, Malaysia

*Corresponding Author

Abstract

During the last decades, freely available GDEMs, such as ASTER, SRTM, and AW3D30, have been widely used in many applications such as for environmental, spatial analysis, research in geomorphology, hydrology, etc. However, these available GDEMs suffer from various limitations. In order to enhance the quality and accuracy of GDEMs, several GDEMs have been merged or reprocessed using a more rigorous method to develop new GDEMs. The advent of these new improvised GDEMs has advanced their applications. Unfortunately, there are very limited studies that focus on the comprehensive and systematic evaluation of the quality of improvised GDEM. Therefore, this study examines the vertical accuracy of three freely available improvised GDEMs (MERIT, NASA, and EarthEnv GDEMs) over the northern region of Peninsular Malaysia using 7757 GNSS points and two reference model, i.e., TanDEM-X DEM 12m resolution and local airborne IFSAR DEM 5m resolution. The accuracy assessments have been performed over three different land covers (urban, non-forest, and forest areas) to evaluate the impact of different land covers on the GDEM's accuracy. Since SRTM DEM is the primary data input in the improvised GDEM, this GDEM is also considered to identify the performance of the new improvised GDEMs. Comparison with GNSS points shows that the accuracy of MERIT DEMs outperforms SRTM DEM and other GDEMs with RMSE of ±2.668m, followed by NASA (±3.656m), SRTM (±5.666m), and EarthEnv (±5.948m). The vertical accuracy of DEM varies with different land cover conditions. Comparison with TanDEM-X and IFSAR DEM shows that all tested GDEMs' accuracy is high over a non-forest area, followed by urban area, and worse over forest area. Overall, the tested GDEM shows only a slight improvement compared to the SRTM. However, these results will help users in selecting the optimum DEM for any application.

Keywords: ASTER, AW3D30, EarthEnv, MERIT, NASA, SRTM, TanDEM-X, Vertical Accuracy

1. Introduction

Digital Elevation Models (DEMs) information play a significant role in numerous applications and research. Typically, there are two main classifications for DEM: digital terrain models (DTM), which exterminate vegetation and buildings from the datasets, and digital surface models (DSM), which incorporate vegetation and buildings in the datasets. They are commonly distributed in gridded data for a certain resolution, where each pixel contains a value representing the local terrain elevation [1]. Nowadays, DEMs remain the essential sources of information for scientific investigations, such as hydrological [1] and [2], natural hazard assessment [3] and [4], geomorphological surveys [5] and [6], landslide identification [7] and [8], etc.

In the early stages, levelling and triangulation techniques were commonly applied to generate DEM information, followed by photogrammetric techniques and remote sensing methods. With the advent of space-borne/airborne sensors technology, DEMs generated using a remote sensing approach through interferometry, and Light Detection and Ranging (LiDAR) techniques have gained high popularity. LiDAR is the most applicable technique to generate DEM, as it has the ability to provide DEM with different areal coverages, resolutions, and high accuracies [9]. Unfortunately, the production of DEM using this technique is typically expensive to operate [10] and unsurprisingly the coverage of LiDAR DEM is still limited. In addition, although LiDAR is capable of fast data collection with dense point distribution, this technology may encounter limitations and become less effective during periods of intense rainfall or when clouds are present [1].

In the past two decades, global coverage DEMs derived using satellite data have attracted widespread attention since the cost of this technique is less expensive compared to the LiDAR [10]. Most of the global DEMs with a medium resolution, such as SRTM and ASTER-GDEM, are freely to download today. Undeniably, SRTM and ASTER -GDEM, which are derived using data from the Shuttle Radar Topography Mission [11] and Advanced Spaceborne Thermal Emission Reflectometer (ASTER) onboard NASA's Terra satellite [12], respectively, are the most widely used in many studies, either in global or regional-scale [13]. These two global DEMs (SRTM and ASTER DEM), which were released in 2003 and 2006, respectively, were among the first DEMs available for free on a global scale [14]. However, the insufficient accuracy of both models makes them unsuitable for local-level studies, in which the accuracy of SRTM-GDEM and ASTER-GDEM ranges from 2.18 to 21.70 m and 4 .56 to 7.10 m, respectively [15] and [16]. Global DEM, namely ALOS AW3D DEM, was freely released to the public in 2015 by the Japan Aerospace Exploration Agency (JAXA). It is derived from the images collected using the Panchromatic Remote-sensing Instrument for Stereo Mapping (PRISM) aboard the Advanced Land Observing Satellite (ALOS) from 2006 to 2011 [17].

This new global DEM is expected to be useful for many scientific types of research. The German Aerospace Center (DLR) has recently produced TanDEM-X DEM, making it the latest addition to the global DEM family. Compared to the available global DEMs, this new global model is expected to provide high-quality geometric resolution and accuracy and be capable of depicting challenging terrain features [18] and [19]. So far, three global datasets with different resolutions have been generated using data collected during the mission spanning from December 2010 to January 2015. The first dataset produced by DLR is in 12m resolution data and is available as Digital Terrain Model (DTM). This global dataset is available for scientific use only and needs a special request to acquire the data. Since the data are not freely accessible, only a few works examine the quality of this DEM [20][21][22][23][24] and [25]. These studies consistently emphasize the superior performance of the TanDEM-X DEM in comparison to other globally available DEMs. Meanwhile, the second dataset, produced by Airbus Defense and Space, is for commercial use and not freely accessible. This global DEM, also known as WorldDEM available both in Digital Surface Model (DSM) and Digital Terrain Model (DTM) [24]. The third global dataset produced by DLR is in 90m resolution, and this dataset is freely available.

Prior to the TanDEM-X DEM-era, the available GDEM at that time had limitations in terms of accuracy, limited observation areas, etc. Their inaccuracies are influenced by the lack of data sources, low resolution, and terrain characteristics. As for SRTM DEM, the well-known DEM product suffers from voids due to shadow-casting [14]. Consequently, the lack of a high-quality global DEM prompted the development of a new DEM through the amalgamation of multiple existing DEM products and their reprocessing using novel techniques and analyses. EarthEnv DEM is among the early improvised GDEM developed by data fusion from a compilation of ASTER GDEM2 and CGIAR-CSI v4.1 products. The fusion of these two products resulted in the creation of a grid of elevation estimates that covers approximately 91% of the Earth's surface, offering improved quality and consistency [26]. Another improvised GDEM is NASA DEM, which was generated by the National Aeronautics and Space Administration (NASA) by reprocessing SRTM data and merging with other data sources, such as ASTER, Ice, Cloud, and land Elevation SATellite (ICESat), and Geoscience Laser Altimeter (GLAS) [14]. The main purpose of NASADEM is to address the issue of void areas in the SRTM DEM and enhance the accuracy of the SRTM DEM data. Multi-Error-Removed Improved Terrain DEM (MERIT DEM) was developed to produce a DEM model that provides elevations of the bare ground by eliminating vegetation [28].

This DEM product was produced by reprocessing the SRTM data and merging other data from ALOS AW3D and elevations from Viewfinder Panoramas (VFP-DEM) [28]. Technically, MERIT DEM represents a global DTM model, which is generally required by researchers compared to the other global DEMs, which represent DSM. Compared to the available GDEMs, these three improvised GDEMs are rarely evaluated. To the best of our knowledge, literature records suggest that there are very limited studies that demonstrate the performance of these three improvised DEM products. Moreover, the accuracy of EarthEnv, MERIT, and NASA DEM in any Malaysian region is yet to be systematically estimated. Therefore, the primary goal of this study is to assess the vertical accuracy of these three DEMs (MERIT, EarthEnv, and NASA DEM) using the reference height derived from GNSS points, TanDEM-X DEM 12m resolution, and airborne IFSAR DEM 5m resolution. In the evaluation process, the effect of different terrain characteristics, i.e., urban area, forest, and non-forest area, on the vertical accuracy of DEM products have also been tested. Since the SRTM DEM is the main source in developing these three improvised GDEMs, this product has also been evaluated using the identical reference height and has been compared with the tested GDEM. It is crucial to identify the quality/accuracy of the tested GDEM by comparing them with the original DEM product, i.e., SRTM DEM.

2. Study Area and Datasets

2.1 Study Area

The assessment of the tested GDEM is performed specifically in the northern region of Peninsular Malaysia, which covers two states, i.e., Perlis and Kedah, as shown in Figure 1. These areas have been selected based on the three main criteria: (1) the availability of dense GNSS points, (2) the availability of a high-precision reference model, and (3) the diversity of the land cover. In the first assessment, 7757 GNSS points covering the study area (Figure 5) are used in the comparison. In the second assessment, three areas with different terrain characteristics, which are forest area, non-forest area, and urban area, are selected as tested areas (Figure 2). For the third assessment, the comparison is conducted using high accuracy IFSAR DEM 5m resolution.

2.2 Global Digital Elevation Models (GDEMs)

In this study, three improvised GDEMs (MERIT, NASA, and EarthEnv GDEMs) have been evaluated together with the well-known GDEM, i.e., SRTM DEM. Details of the characteristics and DEM map for each GDEM are summarized in Table 1 and illustrated in Figure 3, respectively.

2.2.1 SRTM DEM

The GDEM is generated from the images captured by two synthetic aperture radars (SAR) installed on Space Shuttle Endeavour during an 11-day mission in February 2000. The first SAR is a C band system (5.6 cm, SIR-C), which serves to generate contiguous mapping coverage. Meanwhile, the second SAR is an X band system (3.1 cm, X-SAR), which serves to generate data along discrete swaths of 50 km wide [11]). The primary aim of the SRTM (Shuttle Radar Topography Mission) is to generate topographic maps for the region between 60°N and 60°S latitudes and detailed information on the SRTM mission is provided by [11].

Figure 1: Study area over northern region of Peninsular Malaysia (Perlis and Kedah)

Table 1: Details of information about the GDEM

Figure 2: Three selected sites for the comparison of tested GDEM with TanDEM-X DEM and IFSAR DEM

At present, three official versions of the dataset have been made publicly available without charge with several improvements using different techniques and additional data with the goal of overcoming the voids due to shadow-casting. Latest version of this GDEM is version 3 (SRTMV3) which released in 2014, also known as SRTM Plus, where the voids areas are filled using data from ASTER GDEM. Undeniably, this GDEM is the most used in many applications and scientific studies. Numerous studies have been conducted to assess the performance and accuracy of the SRTM DEM with different accuracy results. [11] have summarized that the vertical accuracy of SRTM DEM is about 16 m. However, other studies have reported that the accuracy of SRTM DEM is below 9 m [27]. The SRTMV3 data are available at the United States Geological Survey (USGS) website ( http://earthexplorer.usgs.gov/).

2.2.2 NASA DEM

With the intention to overcome the weakness and improvise the quality of the SRTM DEM, the National Aeronautics and Space Administration (NASA) has invested approximately $1Milion to reprocess the entire SRTM radar data. In the reprocessing and reanalysis, the primary DEM from SRTM was merged with other DEM datasets, which were unavailable at the time of the original SRTM, such as from ASTER DEM, Global Multi-resolution Terrain Elevation Data 2010 (GMTED2010), AW3D30 DEM, Canadian Digital Elevation Data, ICESat, and GLAS were produced.

Since these products are just recently released, limited studies have evaluated their accuracy (e.g., [14] and [29]) and there is insufficient information in the literature discussing their validation in terms of vertical or horizontal errors. An innovative evaluation by Bettiol et al., [29] has found that the precision and accuracy of NASADEM over the Brazilian Cerrado area is lower than AW3D30 DEM. From the comparison results between NASADEM and Brazilian Geodetic Reference Stations, the standard deviation and root mean square error (RMSE) are 8.39m and 8.88m, respectively. Theoretically, NASADEM should be superior to SRTM DEM but the evaluation by Uuemaa et al., [14] has found that the NASADEM exhibited only a marginal improvement when compared to SRTM. Officially released in February 2020, this product can be downloaded from https://portal.opentopography.org/raster?opentopoID=OTSDEM.032021.4326.2.

2.2.3 MERIT DEM

The Multi-Error-Removed Improved Terrain DEM (MERIT) was generated from the integration of SRTM 3 [11] and AW3D-30m DEM data [17] for the areas below and above N60°, respectively. Combination of both DEMs as the baseline DEMs and Viewfinder Panoramas' DEM was used to fill the observation gaps in the SRTM 3 and AW3D-30m DEM [28]. During the process, several errors, such as noises and biases, were removed based on the reference ground elevation from the ICESat laser altimetry [30]. The forest canopy was removed using the global tree density map [31], but the artifacts still remained in the DEM product [32]. Thus, this global DEM is only represented as DTM over forest areas. Hawker et al., [33] and Liu et al., [34] have reported that the error in the DEM was well reduced with an accuracy of ±12m. This data can be downloaded at the MERIT Hydro website ( http://hydro.iis.u-tokyo.ac.jp/~yamadai/MERIT_DEM/).

2.2.4 EarthEnv DEM

The EarthEnv DEM was generated by integrating three GDEMs: ASTER GDEM2 with 30m resolution, CGIARCSI SRTM v4.1, and Global Land Survey Digital Elevation Model (GLSDEM) with 90m resolution. In general, the main objective of the EarthEnv DEM was to provide a high-resolution, high-quality, free, and global consensus product [26]. In the DEM processing and assembly, three (3) processing zones have been established for all the available GDEMs, categorized according to the quality and availability of the input data. The three zones, including the ASTER zone, SRTM zone, and blend zone, were merged to produce a new GDEM with a 90m resolution. Like NASADEM, not many literature records were found discussing the accuracy of this product. In the evaluation of the vertical accuracy of their new GDEM over ASTER and SRTM zones, Robinson et al., [26] have summarized that the product accuracy is almost identical to the ASTER GDEM2 and CGIAR-CSI SRTM v4.1. Moreover, although multi-scale smoothing was applied during the processing, the effect of smoothing is insignificant to the accuracy. However, over the blend zone, the EarthEnv shows significant improvement with an RMSE of 5.362m compared to ASTER GDEM2 (RMSE=15.68). However, another study has reported that the accuracy of EarthEnvi DEM over moderate and rugged topography outperforms MERIT DEM with RMSE of 3.05m and 6.55m, respectively. Details of the EarthEnv processing can be found in [26]. The EarthEnvi DEM product can be downloaded from the EarthEnv website (https://www.earthenv.org/DEM.html).

2.3 Reference Height

2.3.1 GNSS Points

The reference height from LiDAR is usually used to evaluate the accuracy of published GDEM [33]. However, since the LiDAR data are not available over the study area, GNSS points have been used to evaluate the tested GDEM in this study. The GNSS provides high horizontal accuracy; however, the vertical accuracy is 2-3 times lower than the horizontal accuracy. As a result, GNSS height is not commonly used for leveling purposes. Although the ellipsoid height can be transformed to orthometric height using EGM, the plumb lines of both height systems are not identical. In this study, the GNSS points have been obtained from the Department of Survey and Mapping Malaysia (DSMM). In general, all GNSS points which primarily used for cadastral reference marks (CRM) have been observed using the fast-static method. For the evaluation of tested GDEM in this study, a total number of 7757 points have been used and the distribution of the points is shown in Figure 4.

2.3.2 TanDEM-X DEM 12m

The global DEM family has been enhanced with the latest inclusion of the TanDEM-X DEM. This new global DEM is expected to provide the most comprehensive global model of the Earth's surface with unprecedented geometric resolution, precision, and accuracy [18], compared to the available GDEMs (e.g., SRTM, ASTER, AW3D30, etc.).

In general, the DEM information has been generated by using single pass Synthetic Aperture Radar (SAR) interferometry where the corresponding pairs of images are acquired by the twin satellites TerraSAR-X and TanDEM-X, which fly in a close orbit formation and with distances between 300m and 500m of each other to acquire radar images of the Earth surface at high spatial resolution [18]. Currently, there are three models of TanDEM-X DEM with two resolutions have been released.

Three DEM models have been generated utilizing the data collected during the mission spanning from December 2010 to January 2015. The first model is produced by the German Aerospace Center with a 12m resolution. However, this model is only available for scientific use and needs a special request from DLR. The second model is produced by Airbus Defense and Space with a 12m resolution (WorldDEM, 2019).

Figure 3: Four tested GDEMs over the study area

Figure 4: Distribution of GNSS points (7757 points) used for GDEM validation

Figure 5: TanDEM-X DEM 12m (left) and airborne IFSAR DEM with 5m resolution (right)

Known as WorldDEM, this GDEM is available for commercial use only. The third model is TanDEM-X DEM with 90m resolution, which is produced by DLR for non-commercial and is freely available to civilians. It is crucial to highlight that TanDEM-X DEM 12m and WorldDEM are Digital Terrain Models (DTM) in which vegetation and man-made objects have been removed from the datasets, while the TanDEM-X 90 is a Digital Surface Model (DSM). In this study, the accuracy of tested GDEM has been evaluated using the TanDEM-X DEM 12m (Figure 4) over three areas with different elevation characteristics.

2.3.3 Airborne IFSAR DEM

Airborne IFSAR DTM is the second reference height used to evaluate the performance of the GDEM, and this DTM is acquired from Intermap Technologies Malaysia with a 5m resolution. A previous study by Hashim and Mohd (2015) have identified the quality of the Airborne IFSAR DEM using 30 GNSS points. Based on the comparison, the accuracy of IFSAR DEM over flat and undulating areas was 0.497m and 0.841m, respectively. In this study, the IFSAR DTM only covers a small part of the study region (Figure 5).

The comparison has also been performed over three different areas with different elevation characteristics, as shown in Figure 3.

3. Method

3.1 Vertical Datum Conversion

For the comparison, all datasets must be transformed into the same vertical datum. Here, the height of GNSS points acquired from the DSMM and the TanDEM-X DEM 12m are ellipsoid height, while the height of tested GDEM (SRTM, ErathEnvi, MERIT, and NASA DEM) are referenced to the EGM96 geoid. In order to convert the ellipsoidal height, , obtained from GNSS and TanDEM-X DEM 12m, into the orthometric height, , a transformation is necessary. This transformation involves subtracting the geoid height, N (based on the EGM96 geoid model), from the ellipsoidal height, as in Equation 1:

h = H-N

Equation 1

where the geoid height is extracted from EGM96 at 0.1-degree intervals using the F477.F program provided by NGA at http://earth-info.nga.mil/GandG/wgs84/gravitymod/egm96/egm96.html and bilinear method is used for the interpolation [11]. The height extracted from the IFSAR DEM are orthometric heights but reference to the local vertical datum. Therefore, the height from the airborne IFSAR DEM is transformed from the local vertical datum to EGM96 to provide an equivalent comparison. This transformation is performed by computing the geoid height differences (ΔN) between precise local geoid model (N_local) and EGM96 geoid model (N_EGM96). Subsequently, the difference to the IFSAR DEM height (H_IFSAR) is added to produce corrected height (H_corr) as presented in Equations 2 and 3:

ΔN = N _local - N_EGM96

Equation 2

H _corr = H _IFSAR + ΔN

Equation 3

Where the local geoid height information is extracted from the Malaysia precise geoid model (MyGEOID) provided by DSMM.

3.2 Accuracy Assessment

In the comparison between tested GDEM and reference height from GNSS levelling and reference DEM (TanDEM-X DEM 12m and IFSAR DEM), bilinear interpolation is applied to interpolate the DEM elevation at the GNSS positions. For each point, the error is computed by subtracting the reference height (H_refer),from the corresponding value in each global DEM H_GDEMas in equation 4:

ΔH = H_refer - H_GDEM

Equation 4

From the comparison, the positive value of errors indicates that the tested GDEM elevation is lower than the reference height and vice versa. Based on these errors, three statistical metrics are computed: mean error (ME), standard deviation (STD), and root mean square error (RMSE). These metrics are expressed as follows:

Equation 5

Equation 6

Equation 7

4. Result and Analysis

4.1 Vertical Accuracy Using GNSS

In the first evaluation, we examine the vertical height accuracy of MERIT DEM, NASA DEM, and EarthEnv DEM by comparing all models with 7757 GNSS points. To remove any outliers from the GNSS points or DEM, the errors are analyzed based on three standard deviations (i.e., the three sigma rule), whereby any data that contains errors higher or lower than 3σ are identified as outliers. Figure 6 shows the descriptive statistics for each DEM over the study area, and Figure 7 illustrates the vertical error distributions. Based on the distribution of vertical errors, the MERIT DEM shows a symmetric unimodal normal form compared to other GDEMs, which show a positively skewed distribution. The findings indicate that the MERIT DEM demonstrates an overestimation of elevation values within the study area, with a mean error (ME) of 0.825m and the error values range from -7.901m to 7.978m. Among the GDEMs, the vertical accuracy of MERIT DEM outperforms the other three DEMs with the lowest STD and RMSE of ±3.345m and ±2.668m, respectively. It has been expected that the accuracy of MERIT DEMs is better than other GDEMs since this DEM represents the global DTM model.

Meanwhile, NASADEM shows better accuracy compared to SRTM DEM and EarthEnv DEM with ME, STD, and RMSE of 1.476m, 3.345m, and 3.656m, respectively. Followed by SRTM DEM with ME, STD, and RMSE of 3.945m, 4.068m, and 5.666m, respectively. As expected, the agreement of EarthEnv DEM with reference values is the worst. Comparison with GNSS points show the ME, STD, and RMSE of EarthEnv DEM are 4.062m,4.344m, and 5.948m, respectively. This result is unexpected, considering that this global DEM is produced through a compilation of ASTER GDEM2 and CGIAR-CSI v4.1 products, with both global DEMs fused into a quality-enhanced dataset. This could be attributed to the lower resolution (approximately 90m) of these DEMs, which is less compared to the other three GDEMs.

This lower resolution may not accurately capture details in DEMs with low temporal resolution. However, rigorous analysis or experiments should be conducted to confirm this in further studies. In order to find the correlation between GDEMs and GNSS points, four scatter graphs have been plotted, and correlation coefficients are determined (Figure 8). In general, all scatter plots exhibit a perfect fit between the reference and DEM elevation values, indicating a strong correspondence between the two datasets. The results show that the four GDEMs have relatively similar correlation coefficient values. According to the results, MERIT DEM shows the highest positive correlation with reference height for the target area with a correlation coefficient of 0.9943, followed by NASA DEM, SRTM DEM, and EarthEnv DEM with a correlation coefficient of 0.9924, 0.9897, and 0.9874, respectively.

4.2 Vertical Accuracy Assessment Using TanDEM-X DEM

In this section, the accuracy of all GDEMs is assessed by comparing them to the TanDEM-X DEM. Figure 9 shows a statistical analysis of differences between the GDEMs in the three sample areas, which represent different types of elevation and land cover: non-forest area, forest area, and urban area. The results show that EarthEnv DEM and MERIT DEM have a relatively similar average difference, with RMSE of 5.6m and 6.181m, respectively. Meanwhile, the difference between TanDEM-X DEM and two other GDEMs: NASA DEM and SRTM DEM, are almost similar, with RMSE of 4.478m and 4.952m, respectively. As expected, all four tested GDEMs have better performance in non-forest and urban areas compared to the forest area. Besides, all GDEMs also have quite similar RMSE in the urban area and the lowest RMSE in the non-forest area. For the EarthEnv DEM, the mean error (ME) values range from -1.578m to -0.300m, with an average ME of -0.981m. Additionally, the root mean square error (RMSE) values range from 1.923m to 11.531m, with an average RMSE of 5.600m. The EarthEnv DEM outperforms NASA and SRTM DEM in non-forest areas, but both DEMs have low performance in the urban and forest areas. For MERIT DEM, the ME values range from -0.277m to 9.680m, while the RMSE values range from 1.251m to 13.625m . Based on the results shown in Table 3, MERIT DEM is consistent well with the TanDEM-X DEM 12m in the non-forest area with an RMSE of 1.251m. However, it has the highest error among the four GDEMs in the forest area. This outcome indicates that the MERIT DEM exhibits poor accuracy in rugged and densely vegetated areas. Although it has the lowest RMSE in non-forest areas, its average value is the highest among the four GDEMs.

Figure 8: Level of agreement between reference height from GNSS and the four targeted DEMs (a) SRTM (b) NASA (c) MERIT (d) EarthEnvi

Figure 9: Error metric of EarthEnv, MERIT, NASA, and SRTM DEM for each of the three different land cover areas (a) ME (b) STD (c) RMSE

For NASA DEM, the ME values range from 1.167m to 2.157m, with an average of 1.582m. Comparison with TanDEM-X DEM 12m shows that the RMSE values vary from 2.005m to 7.546m with an average of 4.478m, and it is the lowest average value compared to the other three GDEMs.

Although this GDEM shows the lowest average in terms of RMSE, it does not indicate that NASA DEM is the best GDEM. As previously highlighted, this comparison is only to determine the consistency of the tested GDEMs with the TanDEM-X DEM 12m.

For the SRTM DEM, the ME value range from -1.672m to -0.385m with an average of -1.201m, while the RMSE values range from 2.407m to 9.222m with an average of 4.952m. Among the tested GDEMs, SRTM DEM has the highest value of RMSE in the non-forest area but the lowest in the urban area. For this comparison, it can be concluded that the comparison between TanDEM-X DEM 12m with the four tested GDEMs in three different types of topography (non-forest, urban, and forest areas) do not indicate significant differences and have relatively similar RMSE at each sample area. Among the sample areas, the forest area has the highest RMSE value.

4.3 Vertical Accuracy Assessment Using Airborne IFSAR DEM

In the next evaluation, all four tested GDEMs are compared with the IFSAR DEM with 5m resolution at three different classifications, i.e., urban area, forest area, and non-forest area. Figure 10 presents the statistical summary of the comparison between IFSAR DEM, and all tested GDEMs over the three different areas.

Figure 10: Error metric of EarthEnv, MERIT, NASA, and SRTM DEM for each of the three areas (a) urban (b) forest (c) non-forest)

Figure 11: Topography profile of GDEM over three different types of land use: (a) Urban area (b) Non-forest area, and (c) forest area

Based on the results, a conclusion can be drawn that the GDEMs exhibit higher accuracy in urban and non-forest areas when compared to forested areas. In general, the MERIT DEM shows the optimal averages of ME, STD, and RMSE over the three tested areas with averages of -4.197, 2.836m, and 5.288m, respectively. Followed by NASA DEM, where the averages of ME, STD, and RMSE are -5.572m, 2.841m, and 7.820m, respectively. Meanwhile, the EarthEnv performs poorly compared to other GDEMs, in which the averages of ME, STD, and RMSE are -8.743m, 3.785m, and 9.548m, respectively. The most surprising aspect of the results is NASA DEM outperforms the other three DEMs in the urban area, with the lowest RMSE of 2.543m, followed by MERIT DEM with an RMSE of 3.163m. Meanwhile, the EarthEnv DEM and SRTM DEM show similar accuracy but higher RMSE values of 4.744 and 4.794m, respectively. The accuracy of all GDEM over forest area are the lowest compared to the urban and non-forest areas. As expected, the MERIT DEM shows a good accuracy over forest areas than other tested GDEM with an RMSE of 12.087m, followed by NASA DEM with an RMSE of 18.564m. Another notable finding is that the SRTM DEM demonstrates superior accuracy compared to the EarthEnv DEM specifically in forested areas, exhibiting an RMSE of 21.653m. In the case of non-forest areas, a comparison with the IFSAR DEM reveals that the MERIT DEM offers the highest level of accuracy, indicated by an RMSE value of 0.615m. One interesting finding is EarthEnv DEM is the most accurate, with an RMSE of 1.732m, followed by SRTM DEM and NASA DEM, with RMSEs of 1.931m and 2.543m, respectively. To illustrate the spatial variation in the disparities between satellite and IFSAR-based elevations, all the tested. GDEMs have been compared with the IFSAR DEM along a profile across different types of land cover. This profile analysis would provide a visual representation of how the elevations derived from satellite data differ from those obtained through IFSAR technology, highlighting the spatial variability in these differences across various land cover types. Figure 11 presents a comparison along a profile between the four GDEMs at three distinct sites.

The figure visually displays the variations and disparities in elevation values among the GDEMs along the specified profile, allowing for a direct comparison of their performance at different locations. Over the urban area, the elevation of MERIT and EarthEnv are in general higher than IFSAR DEM, and the differences in height values along the profile path are significant. However, the elevation profile pattern of both models is more smother and consistent with the reference height compared to NASA and SRTM DEM. For the NASA DEM, the elevation values along the profile path are close to the reference height, but the profile pattern is irregular compared to IFSAR DEM. Over the non-forest area, the elevation profile pattern of EarthEnv and MERIT DEMs is consistent with the reference height. Along the profile path, the MERIT DEM and EarthEnv DEM elevations are below and above IFSAR DEM, respectively. The elevation profile of NASA DEM is below IFSAR DEM and shows substantial differences with the reference DEM compared to other tested GDEMs. In general, the differences in elevation profile between all of the tested GDEMs and IFSAR DEM over forest areas are not substantial. Besides, the elevations of all the tested GDEMs are above the IFSAR DEM.

5. Discussion

A comparison of all tested GDEMs with GNSS levelling shows that MERIT DEM provides the most accurate DEM compared to other DEMs. It is aligned with the previous study by Uuemaa et al., [14], which found the accuracy of MERIT DEM over Estonian is better than SRTM DEM and NASA DEM with RMSE of 3.01m. It is also almost consistent with the finding by Hawker et al., [33], who estimated the RMSE of MERIT DEM to be approximately 2.32m. However, the accuracy obtained from this study is higher than the expected accuracy range in the previous research, which is 12m for MERIT DEM [28]. Since the MERIT DEM is developed from comprehensive error removal from SRTM DEM, as expected, it will provide accurate DEM compared to SRTM DEM. In general, drawing correlations between this finding and previous research can be challenging due to limited studies that have specifically evaluated the accuracy of the MERIT DEM. Thus, the availability of comprehensive comparative studies for direct correlation is limited, making it difficult to establish a broader context for this finding within the existing body of research. The next-best DEM is NASA DEM. This DEM is also expected to have better accuracy and be the successor to the SRTM. Align with the expectation, comparison with GNSS levelling clearly shows that the NASA DEM offers better accuracy than SRTM DEM, with an RMSE of 3.656m. This result shows that our finding aligns with the previous study by Carrera-Hernández [35]. The study found that the vertical RMSE of NASA DEM is about 3.1m after evaluation with LiDAR DEMs in Mexico. Meanwhile, a different study by Uuemaa et al., [14] has found that the range of RMSE is from 6.39m to 12.08m. However, both studies have agreed that the accuracy of NASA DEM outperforms SRTM DEM. The vertical accuracy of SRTM DEM and EarthEnv DEM are comparable, with RMSE of 5.948m and 5.666m, respectively. Compared to SRTM and other GDEM, the performance of EarthEnv DEM is rarely evaluated. Developed from a compilation of datasets from SRTM CGIAR-CSI v4.1product and ASTER DEM2 and constructed via rigorous techniques [26], EarthEnv DEM is also expected to give better accuracy than SRTM. However, the difference in accuracy between the two DEMs is insubstantial. The accuracy of the SRTM DEM obtained in this study aligns with the findings of González-Moradas and Viveen [21]. Their study, which assessed the vertical accuracy using 139 ground control points (GCPs), produced similar results, confirming the consistency in the accuracy assessment of the SRTM DEM between the two studies with RMSE of SRTM DEM is 5.11m. However, a study by Carrera-Hernández [35] has identified that the SRTM DEM has better accuracies with an RMSE of 3.8m. Overall, the evaluation method utilizing GNSS leveling data has its limitations. This is because the GNSS-derived height represents the elevation of a specific point, whereas DEM provides average elevations in a pixel-based format. For instance, one GNSS elevation point is equivalent to an area of 90x90 meters in MERIT DEM. However, in reality, elevation variations can be observed within a much larger area of 8100 square meters. Therefore, this issue needs to be reconsidered in further studies.

Besides, the type of land cover has a significant effect to the accuracy of DEMs [14]. It is related to the different penetration abilities of satellite sensors. Thus, in the next assessment, all of the tested GDEMs are evaluated based on the different land cover, i.e., forest area, non-forest area, and urban area. Since the distribution of the GNSS leveling points is not well- over the selected land cover areas, two reference DEMs, i.e., TanDEM-X DEM 12m and IFSAR DEM 5m, are used as reference DEM. For each reference DEM, the comparison has been conducted at three different areas.

As expected, the results from the comparison with TanDEM-X DEM 12m and IFSAR DEM indicate that the performance of all tested GDEMs is poor over a densely forested area. However, there is an unexpected result obtained when comparing the GDEMs and TanDEM-X DEM over forest areas. Based on the result, the performance of NASA DEM is superior to other DEMs, and MERIT DEM exhibits the worst accuracy. Theoretically, MERIT DEM should be superior to other models over forest areas because the DEM model incorporates vegetation correction and can be considered DTM, similar to TanDEM-X DEM. The scenario is probably because the performance of TanDEM-X DEM 12m over the targeted area is also unreliable. It could be proven when compared with IFSAR DEM; the MERIT DEM demonstrated superior performance over densely forested areas compared to the other models. However, further study must be performed to confirm this hypothesis. Meanwhile, a comparison with IFSAR DEM shows the NASA DEM exhibits better performance than SRTM DEM. This result proves that the improvement in the NASA DEM data processing is a success as it improves the DEM accuracy. Over the urban area, the comparison with TanDEM-X DEM 12m shows that all tested GDEMs exhibit almost similar accuracy (~ 0.3m), with SRTM DEM indicating slightly better performance than other models.

However, comparison with IFSAR DEM shows significant differences among the tested DEMs, with NASA DEM outperforming MERIT DEM and two other models. It has been expected since the MERIT DEM only applies vegetation correction but not building correction [28]. The most surprising aspect of the result is both comparisons over non-forest area exhibit that the MERIT DEM outperforms other tested GDEM with RMSE of 1.251m and 0.615m. Since this global DEM was developed by combining SRTM3 v2.1 and AW3D-30m v1, the accuracy of AW3D-30m v1 may substantially affect the accuracy of MERIT DEM. Based on the previous study by Uuemaa et al., [14], AW3D-30m offers the most robust and accurate DEM across all regions. Among the tested GDEMs, EarthEnv DEM exhibits the worst accuracy. The first expectation can probably be explained by the low spatial resolution of this model [14], which has a 90m spatial resolution compared to NASA DEM and SRTM DEM, which have a 30m resolution. However, this reason is debatable as MERIT DEM performs well with the same resolution, i.e., 90m.

6. Conclusion

This study provides a comprehensive evaluation of three enhanced GDEMs (NASA DEM, MERIT DEM, and EarthEnv DEM) and SRTM by validating them with GNSS points, TanDEM-X DEM 12m, and IFAR DEM over different land covers in the northern region of Peninsular Malaysia. In general, comparison with GNSS points shows that EarthEnv DEM is the least accurate DEM and MERIT DEM is the optimal DEM compared to others GDEMs across the study areas. Significant improvement in accuracy can be seen where NASA DEM and MERIT DEM outperform SRTM DEM. In addition to the accuracy assessment, further investigation has been carried out to examine the vertical accuracy of the DEMs. This investigation involves utilizing reference height data extracted from both the TanDEM-X DEM and the IFSAR DEM. By comparing the DEMs with these reference heights, a more detailed analysis of their vertical accuracy can be conducted. The results indicate that the accuracy of all tested GDEMs is notably lower in forested areas compared to non-forest and urban areas. This finding suggests that the GDEMs encounter challenges in accurately representing elevation information in forested environments, potentially due to factors such as vegetation cover and canopy interference. Comparison with IFSAR DEM over forest and non-forest area show that the MERIT DEM outperforms other models. However, over an urban area, the performance of MERIT DEM is below than NASA DEM, and it is expected since the MERIT DEM is only corrected for vegetation but not for buildings. From this study, significant improvement in the accuracy of improvised GDEM, i.e., NASA DEM and MERIT DEM, can be clearly seen when compared to SRTM DEM. In addition, this study also shows that the types of land cover greatly influence the accuracy of DEMs, and the user should take this conclusion into consideration.

Acknowledgment

This research was financially supported by the Ministry of Higher Education (MOHE), Malaysia through FRGS 2022 (Reference code: FRGS/1/2022/WAB07/UITM/02/1). The authors would like to extend their gratitude to the Department of Survey and Mapping Malaysia (DSMM) for providing the GNSS point data and IFSAR DEM used in this study. Additionally, special thanks go to the German Aerospace Center (DLR) for providing the TanDEM-X DEM with a 12m resolution as part of the project titled "Towards 1 Centimetre Geoid Model at Southern Region Peninsular Malaysia using the New DEM Model- TanDEM-X" (Proposal ID: DEM_OTHER1156).

References