2.1. Volatile environment

In association with global warming and other vulnerable factors, changing weather, extreme climate events as well as difficult conditions on a jobsite affect directly earthmoving processes. These influences have recently become a serious problem for earthmoving equipment and work safety in particular. This requires a synergetic effort from all stakeholders. Here, volatility implies varying, uncertain, unpredictable conditions, which may cause drastic changes in the operations on a jobsite. To take necessary measures and maintain earthwork performance and safety, more sensory and advisory information is required for the RAS-based stations and units used to control the equipment. A volatile environment may cause much more serious problems when it is also subject to harshness in earthmoving operations as earthworks usually take place under difficult conditions that have been widely referred to as 3D’s, namely Dull, Dirty, and Dangerous. On top of insurance and other remedial expenses, the increase in expenditure for a RAS solution to cope with a volatile environment has therefore imposed another dimension, that is, costs or Dear [Reference Marr8], because the money increases spent to cover the design, planning, and extra measures taking into account volatility factors. The budget limited in project procurement may necessitate a suitable solution of automation among the modes of control to leverage at a compromise.

Apart from 4D difficulties, earthmoving in some domains may also encounter disadvantages known as 3H’s, that is, Harsh, Hostile, and Hazardous, such as in a site with high levels of radiation, lack of oxygen, unhealthy level of pollution, high explosive risk, extreme temperatures and pressures, or in military earthmoving missions [Reference Ha, Yen and Balaguer2]. The application for RAS in such an environment offers a viable solution to satisfy the requirements of safety and resilience, whereby inputs from sensory systems, environmental logistics, and human–machine interfaces are utilized to realize interoperability, operator awareness, control systems, and autonomy oversight [Reference Czarnowski, Dabrowski, Maciaś, Główka and Wrona9]. Harsh conditions may degrade the quality of collected data and performance of controllers when operating in field conditions. For reliable and resilient performance of RAS-based equipment for earthworks against environmental variations, disturbances, imperfect conditions, or ambient changes, the ubiquitous wireless sensing system should allow for a level of redundancy to achieve fault tolerance and system dependability.

2.2. Teaming capacity

In robotics research, the cooperative control of multiple platforms is among the emerging trends. In construction and mining automation, the cooperation of earthmoving machines, considered as a technological challenge, indeed, creates opportunities for research and development in robotics. To address this teaming capacity of autonomous platforms, RAS are playing the key role along with advanced technologies in AI, data science, IoT, and CPSs. A great deal of research and development in defense has been given to the management of fleets of cooperative platforms including ground vehicles. For the mining and construction sector, automation of multiplatform cooperation processes has also been a goal for Original Equipment Manufacturers (OEM). In a recent report [Reference report10], priority is taken in developing collaborative control strategies for construction equipment to carry out the work instructions in a digitalized framework. The aim is to connect all information on the jobsites with the equipment under intelligent machine control to substantially improve safety, environmental friendliness, and in particular, productivity.

It can be envisaged that an automated construction site in this decade will manifest the collaboration of bulldozers, graders, and excavators to undertake ground-based tasks without operators behind the cabin controls; in some cases, the machines will be without cabins. Unmanned aerial vehicles (UAVs) or drones in heterogeneous cooperation with ground equipment will play an important role in smart construction to provide platform-based and site-wide feedback to coordinate onsite platforms, including the possible access to volatile and harsh environments, operating in difficult and harsh conditions, surveying, inspection and monitoring, and most of material handling works [Reference Delgado, Oyedele, Ajayi, Akanbi, Akinade, Bilal and Owolabi11].

2.3. Modes of control and RAS technologies

In this survey paper, the term “autonomous” is attributed to earthmoving machines that can operate without human operators (on board or via remote control) on the basis of a preprogrammed 2D or 3D production model of the job site. Levels of autonomy for unmanned systems can be classified by a framework based on mission complexity, environmental difficulty, and human independence [Reference Durst and Gray12]. The framework covers 10 scales from the lowest to the highest ranging from remote control to full/intelligent autonomy. In noncontextual autonomy potential, these scales can be rendered to nonautonomous, semiautonomous, autonomous, and fully autonomous levels. Although RAS applied to earthmoving tasks may appear to be technologically common across platforms, the level of automation exhibited by each platform can be different, depending on various factors. Focused directly on the platform-centric control process for earthmoving, the five modes of control (MoC) are proposed, namely, (i) functional assist, (ii) tele-operation, (iii) semiautonomy, (iv) full autonomy, and (v) cooperation [Reference Ha, Yen and Balaguer2].

Earthmoving tasks usually depend on the site conditions, and hence, their performance is often affected by changes in the work environment. In face of these changes, robotic platforms have been deployed to help maintain quality, productivity, efficiency, and safety [13]. With RAS applications, distinct advantages achieved can be attributed to the key automated functions of robotic systems as summarized in Fig 1, modified from [Reference Ha, Yen and Balaguer2] to include also AI and communication of higher generations for aggregating data and learning to continuously improve performance of complicated tasks such as multiplatform coordination.

Figure 1.

RAS technologies for earthmoving automation [Reference Ha, Yen and Balaguer2].

The core RAS technologies cover following elements:

• • Perception, to extract and process sensing data from the machinery and jobsite;

• • Localization, to determine the dynamic position of the platform in the environment;

• • Navigation, to plan a safe path to move from an initial location to reach the final destination;

• • Control, to command its actuators to follow desired trajectories and perform the automated tasks from the lowest to higher levels;

• • Communication, to connect and update real-time information of internal and external states of equipment in networks with wireless IoT systems and learning schemes;

• • Coordination and networked robotics, to deploy the fleet of autonomous equipment working in a team.

3.1. Localization and navigation

Although localization and navigation control are among the mature research for mobile robots and autonomous vehicles, their applications to earthmoving automation have to take into account unstructured and cluttered environment [Reference Kim, Chen, Kim and Cho14]. The weather volatile conditions may cause localization and navigation problems for platforms with Global Positioning System (GPS)-denied environments, featureless landscapes, loss of perception on obstacles as well as increasing difficult constraints in path planning. The relative and absolute positions of robotic platforms working onsite have been identified quite accurately by implementing real-time location systems (RTLS) or fusion of them, including GPS, IMU – Inertial Measurement Unit, LiDAR – Light Detection and Ranging, RFID – Radio Frequency Identification, UWB – Ultra-Wide Band, IoT-based sensors, encoder-based sensors, and cameras.

In the functional assist mode of control, GPS-based units can be attached to most earthmoving equipment for localization and navigation. Global navigation satellite system with 3D guidance system (3D-GNSS) can be integrated to robotic excavators for machine control and guidance system. To improve positioning accuracy, a technique called real-time kinematic (RTK) positioning has recently been used with a precision to the centimeter level [Reference Lopesa and Trabancoa15]. This technique uses carrier-based ranging to provide ranges, and therefore absolute positions, which have a magnitude more precise than those available through code-based positioning methods. While high-precision dual-frequency GPS systems are available for earthmoving platforms, since GPS signals are affected by weather conditions, common processes such as digging, loading, hauling, compacting, grading, and paving performed by robotic platforms may not achieve the required accuracy under various weather events.

Unlike indoor localization having mostly fixed features and usually based on reliable design and recognition of visual landmarks [Reference Xu, Feng, Kamat and Menassa16], natural landscapes in the outdoor are drastically various and subject to volatile changes including lighting conditions, weather and seasonal influences, and human activities. Heterogeneous features of artificial landmarks are also uncertain. Therefore, operations of onsite robotic machines require context-aware and augmented SLAM (simultaneous localization and mapping) algorithms. In this regard, visual SLAM has more advantages in exploiting more detail information, for example, in forestry landscape with a sensor rig of RGB cameras, an IMU, and a GNSS receiver [Reference Ali, Durmush, Suominen, Yli-Hietanen, Peltonen, Collin and Gotchev17], or in urban scenery based on a heterogeneous landmark-based navigation approach with monocular vision [Reference Lu and Song18]. Various SLAM algorithms have found a wide range of applications, including robot navigation and augmented reality (AR). It is of interest to use visual SLAM for reconstructing 3D color-mapped point clouds (collected from LiDARs or laser scanners) to reflect an uncertain terrain and identify onsite static and dynamic obstacles [Reference Kim, Chen and Cho19]. Visual SLAM is considered as efficient in a fast reconstruction of a dense 3D point cloud model of the earthwork site with an UAV platform combined with an embedded graphics processing unit [Reference Huang, Yang, Tan, Peng, Tao and Peng20]. The semantic SLAM combines 3D mapping and semantic segmentation to enable the robotic equipment to acquire information of a large-scale outdoor environment and facilitate reasoning ability for language-based human–robot interactions [Reference Zhao, Mao, Ding, Ren and Zheng21].

Localization in a GPS-denied environment is typically based on the sensor fusion approach to take advantage of both relative and absolute position measurements. In earth works, RFID, encoder, UWB, IMU, and IoT-based sensors are fused in relative positioning. The relative positioning method is widely used because it allows for high sampling rates and is inexpensive. However, it accumulates error over time if no appropriate compensation measure is taken. For this reason, an IMU is often used with intermittent absolute position measurements to reset the error after a certain period of time. The main advantage of absolute position inputs to an inertial navigation system is their independence from previous estimations, which tends to reduce accumulated errors. Its limitations include complexity of implementation and dependence on the environment structure. Besides, for low-cost solutions, the RFID technology [Reference Wu, Wang, Chen and Kim22] and IoT-enabled systems [Reference Louis and Dunston23] can be attached on fixed objects, workers uniforms, and mobile equipment to enhance the proximity detection and safety assurance. Absolute positioning can be achieved in a GPS-denied environment via a map-matching technique. This approach uses geometric features of the environment to compute the location of the vehicle. The features can be lines and points that describe walls and corners or specific shapes such as rectangles or triangles depending on the object structure. For localization and navigation of earthmoving platforms, the commonly applied GPS can be alternated by the fusion of other approaches such as odometry, visual patterns, UWB, and the emerging deep learning neural networks for landmark detection [Reference Nilwong, Hossain, Kaneko and Capi24] to improve accuracy and reliability.

For earthmoving automation, construction machines have predominantly utilized vision-based technologies, where cameras play an important role. For most surface earthworks or open-pit mining, where spaces allow for good lighting conditions, cameras remain an effective tool for localization and navigation with robotic vision. Indeed, for reconstructing 3D point cloud, stereo cameras having two or more lenses with a separate image sensor or film frame for each lens can achieve better efficiency than LiDAR regarding the postprocessing time, power supply, and data interfaces for scanning a terrain [Reference Sung and Kim25]. Thermal cameras, being able to recognize and capture different levels of infrared light, have an advantage for localization and navigation in the poor visibility condition. They help improve quality management of earthworks such as for nighttime operations such as to enhance the pavement durability [Reference Nevalainen and Pellinen26]. Multispectral images, for example, obtained from RGB and thermal cameras and using semantic segmentation [Reference Ha, Watanabe, Karasawa, Ushiku and Harada27], can provide a better accuracy in real-time localization and navigation for autonomous vehicles operating under poor visibility or adverse weather conditions. A single camera using a complementary metal oxide semiconductor (CMOS) sensor can also be used for multispectral imaging to identify various features, especially when mounted on UAVs [Reference Morales, Guerra, Horstrand, Diaz, Jimenez, Melian, Lopez and Lopez28].

In underground mining, earthmoving equipment faces more severe 3H’s difficulties. Specifically, unclear landmarks, high humidity and temperature, communication ineffectiveness, GPS unavailability, muddy, and rough terrain inside tunnel cause equipment tasking unreliable, which degrades significantly the performance of tele-operation and camera-based tracking techniques [Reference Dadhich, Bodin and Andersson29]. Therefore, laser scanning technologies with enhanced precision techniques are in favor for underground localization and navigation. In ref. [Reference Ren, Wang and Bi30], a Generalized Iterative Closest Point (GICP)-based algorithm is applied to obtain a 3D point cloud of LiDAR-collected data to enhance SLAM resilience in confined spaces. Thermal camera can be fused with LiDAR sensor for autonomous exploration in underground mines [Reference Dang, Mascarich, Khattak, Nguyen, Nguyen, Hirsh, Reinhart, Papachristos and Alexis31]. A multisensor localization technique combining LiDAR and cameras has been recently developed to improve robustness of autonomous operation within two operational Australian underground hard-rock mines [Reference Jacobson, Zeng, Smith, Boswell, Peynot and Milford32]. Toward full autonomy of equipment in overcoming such hazards as rockbursts, squeezing and creeping rockwalls, ditches of water drainage, as well as airborne dust, the extrication of humans from underground mines to enable improved safety and efficiencies still remains the top priority.

3.2. Sensing and perception

Sensing and perception in RAS-based earthmoving aim to develop and deploy accurate measurement systems for real-time monitoring of dynamical parameters inside and surrounding the equipment (e.g., position, speed, acceleration, location, images, temperature, humidity, moisture, pressure, frequency, etc.) as addressing the state identification problem [Reference Sun, Baek, Hwang, Lee, Lee, Jang, Ji, Han and Han33]. The collected information is essential to feed to a sensor-based controller to adapt any changes of environment and increase the contextual awareness in different and complex tasks at jobsites [Reference Jiang and He34]. For instance, a machine guidance system for a bulldozer was developed [Reference Sun, Kim, Lee, Lee and Han35], based on the fusion of sensors such as IMU and RTK-GPS, to provide navigation assistance and to increase earthwork productivity via estimation of the blade pose. Besides, machine’s pose estimation [Reference Soltani, Zhu and Hammad36] is needed to maintain performance during weather events by enhancing perception with multiple sensors attached on platforms. This technology requires the advanced methodologies to fuse the internal parameters and external information for the relative and absolute position and orientation of the platforms and its attachment. In dealing with environmental volatility, the estimated poses of heavy equipment are critical not only to reduce the damage of collision but also to increase the precision of cooperation and autonomous level. The integration of multiple sensors such as factory-installed GNSS antenna and receiver, IMU or enhanced IMU, and hydraulic cylinder stroke sensors on an equipment can constitute an intelligent sensing and controlling framework [37], tele-operation relies mostly on quality of wireless communication system for safe operation.

In the tele-operation mode of control, vision-based and auditory sensors increase the environmental perception capacity with the use of multiple cameras for streaming live videos from the sites to the control center. Since the regular camera provides 2D images, information obtained is insufficient to cope with uncertain position and orientation of complex and high degree of freedom actuators, in particular in volatile weather conditions. Accordingly, the 3D pose estimation by using stereo vision and its fusion with other RTLS offers a prevalent approach [Reference Soltani, Zhu and Hammad36]. Recently, 3D point cloud images reconstructed by the laser scanning methods to provide reliable data have been also promising because this technology is less vulnerable to operational conditions, for example, sunlight, darkness, or harshness on the field [Reference Naghshbandi, Varga and Hu1].

As safe and efficient tele-operation requires a deep perception of site environment through a variety of sensing devices, performance of remote monitoring and control relies on network communication and data throughput. However, the communication speed and connectivity are subject to field conditions, and hence, directly affecting the quality of service (QoS). Therefore, the specialized design of communication systems at construction sites and underground mines need to consider not only the reliability, speed, and coverage of networks but also security and effects caused by long-distance tele-operation [Reference Dadhich, Bodin and Andersson29]. These may include jitters, data packet loss, and blackout but most serious is time delay. To mitigate tele-robotic time delay, control approaches have been proposed, covering model-based, adaptive-based to predictive control techniques. Recent prediction approaches based on machine learning and neural networks are promising in addressing time delay to this open problem [Reference Farajiparvar, Ying and Pandya38].

3.3. Control and planning

For automated earthmoving, to improve the ultimate work capacity and task efficiency on a jobsite in face of environmental volatility, research and development on modeling, control, and planning remain active. For ground-based mobile machines, specific difficulties caused by harsh environments, weather changes, or dynamic conditions may include slippage due to heavy rain, snow, flash flood, or operations on various ground types like rock, sand, mud, soft soil, and uneven terrain.

For autonomous excavation, in addition to known compliant control strategies like impedance or admittance control, the task planning and control can also be integrated in generation of dig cycles regardless of the soil composition by using a bucket tip force–torque trajectory. The bucket forces and motion of the excavator arm are controlled using hierarchical optimization along with a large-scale iterative planner for consecutive execution of single digs to form the desired in-ground geometry. This strategy can handle different types of soil to deal with the tool–soil interaction for different terrains using the Vortex simulation package [Reference Jud, Hottiger, Leemann and Hutter39]. Field validation of an iterative learning-based control algorithm for autonomous excavation has recently been reported in ref. [Reference Fernando, Marshall and Larsson40] with a 14-ton capacity load–haul–dump (LHD) machine for two different types of excavation materials: fragmented rock and gravel. With iterative learning, the admittance control parameters can be automatically updated based on the fill weight error at the end of each excavation pass.

Since possible incidents due to unpredictable changes in the environment can take place on a jobsite, earthmoving equipment should adopt appropriate measures based on environmental sensing. As such, a RAS-based controller has to be equipped with the capability of observing the work environment and responding to its changes. Recently, an architecture combining perception using LiDAR and cameras, along with learning-based and optimization-based planning has been developed for a 7.5-ton excavator to perform autonomous tasks of material loadings continuously in 24 h [Reference Zhang, Zhao, Long, Wang, Qian, Lu, Song and M.41]. In industry, some OEM have incorporated the environmental perception function in their remote controllers, particularly for dozers and loaders [Reference Staff42] or implemented RemoteTask systems for skid-steer, multi-terrain, and compact track loaders, where the machines can be controlled at a safe distance as far away as 300 m in dealing with difficult environments. An automated bucket-loading system, the Autodig, has been successfully applied to improve bucket fill and cycle times to boost productivity in mining and construction, with a high tolerance of uncertainties in the soil and working conditions. To be able to handle a vast amount of data coming out from the adaptation to changes, the 5G mobile communication technology has been recently applied to not only significantly reduce communication latency [Reference Bennink43] but also support data processing for robust systems of connected sensors for weather-related events to mitigate incidents on a jobsite [Reference Yang and Hua44].

As previously mentioned, the smart construction framework could enable the IoT connection for an earthwork site and its 3D visualization. The service features include site topography and design, information of area, shape and volume, and especially the intelligent machine control of earthmoving equipment. In terms of control, the intelligent machine controlled (iMC) technology developed by Komatsu for their dozers and excavators is well integrated with 3D machine control components at the factory level for sensing stroke, load, and other autonomous actuations based around the 3D GNSS and IMU as well as its enhanced version (IMU+) [37]. The iMC system allows for the RAS-based equipment in dealing with difficulties arising from changes in the work environment for various earthwork types to ensure accuracy and efficiency.

3.4. Skid-steering and slippage

For mobile robots and autonomous vehicles, the integration of highly accurate sensors and their fusion are useful for navigation control. However, slippage remains one of the main issues associated with platforms navigating on a jobsite particularly by skid-steering. Slippage is usually associated with skid-steered platforms in robotic earthmoving, wherein high torques are required to perform a curvilinear motion. As a consequence of slippage, those platforms may be unable to follow trajectories generated by the motion planners. To incorporate this issue, dynamic models can be developed to combine the machine’s wheel or track slip with mechanics of the terrain. The models can be obtained by using estimation algorithms such as a system observer or an extended Kalman filter (EKF). For modeling and calibration dealing with skid-steered platforms, an online leaning framework has been proposed by combination of the slip and terramechanics models of wheel–terrain interaction using Kalman filtering (for kinematics updating) and neural networks (for dynamics updating) to achieve the high performance in curvilinear motion with improved energy efficiency [Reference Ordonez, Gupta, Reese, Seegmiller, Kelly and Collins45].

For the wheeled mobile robots, slippage may occur to the wheel’s effective radius variation due to heavy load, tire wear issue, and an extra component from a significant change in the slip model parameters under extreme weather conditions. For example, heavy rain was the cause of a collision between two autonomous trucks [Reference Casey7], when the unloaded truck traveling at about 14 km/h lost traction and slid to the other, loaded and traveling at 27 km/h. Mobile wheeled platforms, such as front-end loaders, tractorshovels, pay-loaders, high lifts, and skip loaders, are all subject to slippage on snowy or wet roads and rough terrain, causing the tires to wear out faster and leading to less tractive force. Passively, the torque proportional differential can be replaced by the limited slip differential to deliver effective driving force to both wheels for enhanced grip and less slippage during travel. Actively, RAS-based traction control can be developed to deliver the highest possible tractive force by using sensors’ data and mathematical modeling of the tire–ground interaction [Reference Alexander and Vacca46].

For tracked vehicles, longitudinal and lateral slippage occurring during straight maneuvering can be derived by approximating the terrain-mechanics slip model or using a transient shear model for the slip and track contact on hard ground [Reference Özdemir, Kılıç and Ünlüsoy47]. In ref. [Reference Lu, Xiong and Guo48], a slip model based on the instantaneous centers of rotation is developed based on position measurements and an EKF. The proposed model was validated in experiments on a 13.6 ton tracked platform equipped with a RTK GPS. Tracked slippage may cause inaccuracy in the platform localization, especially when traveling on loose slope. In ref. [Reference Yamauchi, Nagatani, Hashimoto and Fujino49], slip-compensated odometry, which applies the slip model to the kinematics of a skid-steering vehicle, is proposed for evaluation of the slip estimation of the interaction between a tracked vehicle and an unknown terrain. Although evaluation of the slip model and slip-compensated optometry proposed therein was conducted on a small tracked vehicle on an indoor sandy slope, the promising results can be extended to a robotic excavator or dozer with skid steering. For platform’s turning, longitudinal slippage can be derived from an empirical equation for the relationship between slip ratio and input velocity, and lateral slippage is obtained from a regression function.

3.5. Tool–soil interaction

In robotic earthmoving, an automatic control system developed for executing an earthwork task (digging, bunding, tilting, trenching, profiling, or grading) should take into account of the tool–soil interaction forces arising from the contact in real time. They represent reactions to the driving tool and vary in accordance with the soil properties characterized by the soil type, humidity, and other terramechanic parameters [Reference Marin50]. Thus, from the interaction in the contact phase of the machine’s attachment, the digging forces are highly nonlinear and depend on the soil composition (sand, rock, gravel, mud, etc., or a mixture of them) and its properties, soil volume, tool shape, and working cycle. Interactions between equipment tools and soil are important for autonomous operation of an excavator. Small variations in soil properties may cause significant changes in its static and dynamic behavior, particularly in an inhomogeneous medium. This explains why weather changes or other volatile factors can affect directly work productivity of the earthmoving process.

In the soil cutting and pushing process, the edge of the end effector first penetrates the soil up to a certain depth and the machine then starts cutting, shearing, and pushing the soil. During this interaction, the blade experiences enormous resistance owing to friction, cohesion, and adhesion between the tool and soil, and the soil and ground. The forces acting on the tool vary in a complicated manner that may negatively impact the equipment performance. In the literature, the resistance or “draft force problem” has been tackled either experimentally, or by developing analytical models and using numerical methods [Reference Haeri, Tremblay, Skonieczny, Holz and Teichmann51]. Methods used for parameter estimation in different tool–soil interaction models have included multilayer neural networks and swarm intelligence [Reference Bui, Nhu and Hoang52], discrete element modeling [Reference Narayanan and Bhojne53], the finite element [Reference Marin50], and the spectral element method [Reference Boudaa, Khalfallah and Hamioud54].

Soil parameter estimation enables prediction of interaction forces by changing tool parameters such as the depth of cut, the attack angle of the tool, and digging strategies [Reference Reginald, Seo and Rasul55]. Recent effort has incorporated the soil–tool interaction also in designing an optimal bucket to achieve the highest productivity of the tool and the platform as a whole [Reference Kenarsari, Vitton and Beard56]. However, to achieve accuracy in modeling the soil behavior, more complex rheological models are required, often at a larger computational time and hence may be infeasible for application in a real-time control. Therefore, in addition to the effective parametrization of the soil–tool interaction model in dynamics of the control process in real time, another approach is to treat it as a disturbance and design a robust strategy for control.

As the distribution of pressure and force is nonuniform when the equipment attachment tips interact with different soils and loads, tactile sensors using the piezoresistive, piezoelectric, optical, capacitive, or elastoresistive effect can provide measurements of the medium reaction with high accuracy for estimating soil parameters or predicting tool wear. Information of the terrain chracteristics from tactile sensing can assist with machine traversability on different terrain types to improve navigation on earthworks sites [Reference Kenarsari, Vitton and Beard57]. In ref. [Reference Mattetti, Varani, Molari and Morelli58], soil pressure was measured with 10 tactile sensors attached on the plough. Results obtained are useful for validating numerical models of the soil cutting process leading to increase task efficiency. By merging force sensing (1D) with LiDAR intensity information (1D) and other proprioceptive and exteroceptive sensors for 3D ground map, the 5D map generated can be used to estimate the bucket-load volume and effectively monitor the excavation progress [Reference Rasul, Seo and Khajepour59].

Under the influence of a volatile environment, the tool–soil interaction would become more complicated and unpredictable. Remote sensing methodologies could therefore provide dynamic information [Reference Yuzugullu, Lorenz, Fröhlich and Liebisch60] to compensate for such variations, and robust controllers could mitigate the disturbances impact. Solutions to the tool–soil problem in earthmoving often come from control strategies at both low level and high level. For example, in robotic excavation, a digging strategy can be rendered to track the piston position and ram force of each hydraulic cylinder for the axis control of the boom, arm, and bucket. The idea can be elaborated in compliant control schemes such as impedance or admittance control, to adjust the dynamic relationship between the end effector position and the exerting force, considering the soil contact. Overall, to achieve the effectiveness of automated earthmoving, a control strategy has to deal with soil parameter identification [Reference Bennett, Walawalkar, Heck and Schindler61] and force prediction [Reference Zhao, Wang, Zhang and Luo62]. In this regard, the control problem considering the tool soil contact for earthmoving under volatile conditions remains open for further research in both academia and industry.

3.6. RAS-enabled functions

Since earthmoving often involves dull tasks such as compacting, loading, and dumping materials, RAS-based machines can take over the repetitive and tedious operations to reduce the labor and time expenses. In road construction, a grader can be equipped with a functional assist unit to ease the operator’s control of the machine’s blade and ripper functions in order to achieve high performance of grading on various difficult environments and hence to improve its productivity. For example, the equipment reported in ref. [Reference America63] can handle such functions as Grade control, TOPCON 3D machine control (3DMC) plug-and-play, and rearview monitoring. Especially in dealing with environment changes, the grader can have a full-color high-resolution monitor with Ecology Guidance, as well as pioneering KOMTRAX telematics system and monitor to provide machine metrics, fuel consumption, and plus performance information.

Indeed, to cope with the difficulties arising from uncertain and unpredictable factors of the environment, RAS-based technologies have been developed to provide assist functions on equipment used in the earthmoving industry. As introduced in ref. [Reference Europe64], a versatile supportive and advisory system can offer multiple productive services. These cover common functions for earthworks, as illustrated in Fig. 2, including:

Figure 2.

Functional assist system on Volvo equipment [Reference Europe64].

• • Load Assist: to optimize load cycles via a set of smart apps for boosting the efficiency with such features as (i) a dynamic load weighing system for the bucket’s load to enable safe operation and avoid overloading, underloading, reweighing as well as waiting times and (ii) real-time guidance to operators in actions and improvement of their operation technique.

• • Compact Assist: to provide in real time all onsite information necessary for soil or asphalt compaction such as compactor’s pass, temperature mapping, and density calculation to ensure consistent quality and avoid wasteful over-compaction.

• • Dig Assist: to improve accuracy and efficiency in all excavation tasks such as digging, leveling, trenching, grading, or profiling from an in-cab tablet. The Active Control system can assist in controlling motion of the boom, arm, bucket, swing, bucket’s tilt, and attack angles, as well as limiting the machine’s swing, the depth and height of the cut. The Co-Pilot system can assist in locating the construction equipment to centimeter accuracy with sensors and the RTK and GNSS, as well as importing external 3D design plans in complex projects.

• • Haul Assist: to optimize haul cycles with on-board weighing and onsite monitoring map for hauler traffic in real time.

• • Pave Assist: to ease with the heavy work of paving in a harsh environment with a Material Manager identifying the machine throughput efficiency and keeping a record of calculated distance and tonnage paved, paved area and CO $_2$ emissions, the volume of delivered asphalt and its thermal profile, as well as the weather view.

• • Report Assist: to provide comprehensive reports on productivity, fuel efficiency by using telematics data for improving the efficiency and uptime of the fleet, and thus, maximizing the task capacity and work efficiency of the machines’ operations.

Those functions can help an operator dealing effectively in one way or another with difficulties caused by a volatile environment. For example, as changes in the soil mechanic properties may affect loading or hauling, real-time guiding advice and automated weighing information can help an operator adjust or give input to control strategies. Likewise, various excavation tasks in face of unpredictable conditions can be adjusted with the Volvo Dig Assist service from Co-Pilot system. Moreover, information of environmental parameters, weather conditions along with machine characteristics can also be collected on-board and even available for reporting purposes.

In addition, Volvo also provides its Simulators for training on authentic scenarios to manage skills and to support evaluation and development with virtual data analysis for monitoring project progress, especially with any presumed weather conditions in dealing with volatile environments. Similar products are also available in mining and rock excavation, including the Automine, an automated loading and hauling system for underground hard-rock mining, and data-driven advanced analytic systems for improving maintenance, safety, productivity, and operational services of mining and rock excavation equipment [Reference Mining65]. Indeed, the use of remote monitoring integrated with advanced analytics and cognitive technologies for fusion of equipment data from a range of resources can automatically analyze the patterns to increase mining performance and decrease the cost per tonne with RAS-based and information technologies. Environmental awareness from onsite data measurements and analytics can effectively assist a tele-operator to manage in real time the jobsite under varying work conditions.

In dealing with volatile environments, it is essential to quantify the weather-related impact, tracking all aspects of the machine, earthworks, and the environment in accordance with automated capturing and predicting weather profiles. Field management for robust earthmoving may rely on multiple third-party weather services such as World Weather Online’s weather API (Application Programming Interface), AccuWeather, OpenWeatherMap, Weather Underground, and Weather Forecast from the local Environmental Protection Authority (EPA). These services can be integrated via APIs to allow RAS-based platforms to take necessary corrective actions either by controlling the equipment at the low level of actuators, job guiding, task re-planning or rescheduling at the higher level, and safety/difficulty reporting. The Weather View in multiple productivity services mentioned as part of the various functional assists in Volvo Co-Pilot system, as illustrated in Fig. 3 for paving tasks, can provide up-to-date information for better planning and adapting with earthworks against weather events.

Figure 3.

Co-Pilot system featuring (a) thermal profiling, (b) weather view, (c) material manager, and (d) jobsite view [66].

In terms of handling weather impacts, it is promising to explore frameworks of IoT-enabled collocated low-cost sensing networks, such as the one used for monitoring urban particulate matter emissions on a construction site [Reference Nguyen, Nguyen and Ha67]. This approach can offer an inexpensive solution to provide microclimate information, to gather trusted monitoring data, and forecast meteorological trends for a local site for earth works. With increasing demands for automation in earthmoving for the construction and mining sector, the research and development for application of RAS-based technologies remain to be active in the next decade, given changes in climate and the environment.

4.1. Multi-agent systems

Multi-agent systems (MASs) are computerized systems or machines that comprise multiple interacting intelligent agents. Given their autonomous, cooperative, and learning attributes, MAS are suitable to address the essential problems of “collaboration and consensus” among stakeholders in construction projects. This can be explained by the three prominent attributes of MAS, namely (i) autonomy: the ability of sensing, self-decision and response of each agent in the system; (ii) cooperation: agents collaborating to gain an objective that an individual one could not acquire; (iii) learning: agents self-evolved, adapting with environment and enhancing performance. In addition, taking advantages from software engineering and AI, the MAS framework facilitates the system capability of dynamic planning and scheduling to reduce project’s cost, completion time, and resources.

With the dynamic environment of an earthmoving site scattering with material, machines, wastes and humans, planning of mobile platforms can be achieved from a MAS-based solution for a consensus problem [Reference Bu, Yu, Hou and Qian70]. By considering machines as agents, MAS may be useful for collision avoidance between RAS-based platforms. Indeed, collisions between machine–machine or machine–obstacle remain a challenging issue for safety of autonomous cearthmoving. As mentioned above, a collision of two tele-operated haul trucks on a mine at Pilbara, Western Australia occurred due to a communication drop-out from heavy rains [Reference Casey7]. This indicates the need of considering volatile environment in a RAS-based research and development for the control and communication of cooperative machines. In this regard, for multi-robot systems, a generic method, called $\epsilon$ -Cooperative Collision Avoidance ( $\epsilon$ CCA), has been proposed in ref. [Reference Alonso-Mora, Beardsley and Siegwart71], based on an optimization in the space of control velocities in which moving obstacles are decision-making agents. Soft computing approaches are also promising for MAS as preliminary reported in ref. [Reference Huang and Liu72] based on human’s inner attention mechanism as a communication and multi-modal information fusion strategy, or in ref. [Reference Stolfi, Brust and Bouvry69] based on coevolutionary genetic algorithm for heterogeneous teams of multiple uninhabited vehicles.

4.2. Cooperative mapping and formation control

Mapping is the process of recreating the virtual environment for autonomous machinery to operate and interact with the real-world environment from its sensing information. The recent development of swarm robotics has expanded to an architecture known as multi-robot simultaneous localization and mapping (multi-SLAM) [Reference Kshirsagar, Shue and Conrad73]. While the efficiency and processing time can be improved, implementation of the architecture is restricted due to constraints such as communication bandwidth, memory requirements and problems faced during map merging and co-ordinate transformation. In construction, where different machines can share the work on a jobsite, various platforms can also cooperate automatically using RAS technologies to fulfill complex tasks. For this, a promising hierarchical architecture for a group of heterogeneous mobile robots was proposed in ref. [Reference Masehian, Jannati and Hekmatfar74] using information space and sensor models. The algorithm can fuse limited sensory information, interrelate dissimilar data obtained by individual heterogeneous robots, and allocate various exploratory tasks to each of them in order to complete the environment map.

To obtain topographic information of earthwork sites, UAVs can be used to capture high-resolution images that can be used for 3D reconstruction. To improve model accuracy, the fusion of photogrammetry, laser-scanning, GPS and other sensory data can create a better point cloud map [Reference Moon, Chung, Kwon, Seo and Shin75]. One advantage in using UAVs is that they can cover obstructed or inaccessible areas where ground vehicles cannot reach. The heterogeneous combination of ground vehicles and UAVs can enhance the real-time tracking of machine locations and trajectories. For instance, multiple-session mapping and onsite object tracking with Visual SLAM using UAVs can extend the maps with fast reconstruction of the 3D point cloud [Reference Huang, Yang, Tan, Peng, Tao and Peng20]. From these merits, drone 3D mapping has been included in Komatsu’s smart construction framework [76] integrating with IoT and 3D visualized technologies to capture the 3D point cloud data and map the updated earthmoving sites, which are inputs of multiple tasks such as progress tracking, stockpile volume managing, online collaborating and communicating between the autonomous machines.

The control of unmanned vehicle teams has been an active topic in robotics research. This involves the initialization, establishment and maintenance of a group of mobile robots moving in a desired geometrical shape or formation. It is known that formations of automated vehicles can been applied in intelligent transport systems to save energy and enhance safety. Formation control has attracted many researchers with majority of works focusing on the leader-following methodology based on behaviors or models of multiple robots. In ref. [Reference Lee and Chwa77], a decentralized behavior-based algorithm has been proposed where the robust formation is achieved by maintaining the distance and angle of each robot toward the leader robot without information of its location. To avoid collisions, the heading angles of all robots are determined by generating an escape path along which a robot can move toward a safe layer. The leader follower technique in robotic formation was used to form a convoy of trucks equipped with a laser-based sensor (LiDAR) used for maintaining the distance clearance and demonstrating the V2I concept [Reference Lavery78]. One of the key issues in formation maintenance and path planning is to resolve conflictory constraints between the leader and the followers, for which the optimization and algebraic graph theory frameworks [Reference Ebel and Eberhard79], the Nash–Stackelberg equilibrium in game theory [Reference Ji, Xue, Ha, Luo and Zhang80] and particle swarm optimization-based algorithms [Reference Phung and Ha81] offer promising solutions.

Teaming technologies for autonomous earthmoving equipment are evident as can be seen in autonomous loading by a front-end loader or an excavator and a dump truck, for example, [Reference Schmidt and Berns68]. Indeed, the cooperative operations of equipment obviously add to improved task efficiency on an earthmoving site, giving more room for effort to address requirements on sustainable development. Therefore, this theme is being pursued by leading equipment manufacturers. Figure 4 shows a team of Caterpillar equipment, autonomous dump trucks, surface shovel, and bulldozer working together on a surface mine. Semiautonomous cooperative control technologies are expected to reach higher levels of maturity in the future, given the need of machine coordination in smart mining, as reported by Komatsu, which has been recognized as international leaders of managing climate change [Reference report10].

Figure 4.

Caterpillar autonomous equipment teaming on a surface mine [Reference Mining82].

The growing interest in driverless multi-vehicle networks has resulted in may research works focusing on estimation, perception, planning, and control to support cooperative and coordinated vehicle autonomy. In construction and mining, the coordination and distributed control of heterogeneous machines on a jobsite have stimulated many OEM in applying advanced technologies in robotics and automation, sensing and communication as well as data analytic and AI to be able to deploy a fleet of autonomous platforms to share a complicated or iterative task.

4.3. Cyberphysical systems, communication, and AI

The concept of integrating communication, control, and computing for systems to operate in real-time along with physical machines in feedback loops, whereby physical processes interact with computations to maintain work performance in face of the changes of environment is known as CPSs. It represents a holistic architecture of virtual models, hardware (equipment, sensors, and actuators), and a communication paradigm such as the cloud-based IoT to deal with coordinating and managing large-scale projects. Virtual objects can be generated from Building Information Modeling (BIM), which is a set of new processes to design, plan, and construct buildings [Reference Tang, Shelden, Eastman, Pishdad-Bozorgi and Gao83].

For earthmoving, a cloud-based CPS framework can consist of independent and heterogeneous components (e.g., sensors, actuators, gateways, and server) and dynamic composition of their functionalities, along with smart nodes mounted on various equipment to monitor wirelessly their operation. The framework can allow for remote management of multiple big projects simultaneously in different locations, for example, highway construction and photovoltaic power plant construction. It consists of two main parts, an onsite IoT network and a cloud-based system with user interface for the project manager [Reference Bucchiarone, Sanctis, Hevesi, Hirsch, Abancens, Vivanco, Amiraslanov and Lukowicz84]. In volatile conditions, the IoT can support communication, sensing, and control for platforms with onsite digital intelligence. As an example, the internal IoT-based sensors automatically detect and send alert signals related to equipment states for active maintenance and replacement before any problem could occur. In combination with RFID technology in asset management, the transportation of material between different sites can be updated in real time, and the response of operators during logistic is also monitored to mitigate potential collisions or unexpected events [Reference Khokale85]. To facilitate the teaming capacity, the machines make themselves recognizable, aggregate information, and share data with others to perform automated tasks. As such, the whole system can enhance reliability and dependability to cope with changes in the environment.

Multiplatform teaming would require system-level technologies to handle to the complex interaction between earthmoving equipment, workers, and materials or objects in cooperative works [Reference Naghshbandi, Varga and Hu1]. Stand-alone activities of each RAS-integrated platform of the team would need to efficiently fulfill the task decomposition in earthwork coordination. Hence, the context awareness of autonomous machines on the site needs to be secured for harmonious cooperation to improve productivity and avoid the unexpected collision and accident. For this, the wireless communication between the equipment based on IoT networks is increasingly adopted to schedule and manage the cycle operation of equipment [Reference Jiang and He34] owing to advantages from 5G technologies [Reference Reja and Varghese86]. To achieve error-free wireless communications for cooperation of group of equipment, the intelligent excavation system (IES) proposed in [Reference Kim, Lee, Seo and Kamat87] suggests the use of multiple hardware and software modules which interact each other with diversity of communication protocols (e.g., TCP/IP, SCI, etc.). This framework is managed by a data communication manager and an equipment data management system to coordinate large amount of data from micro (sensing and control) to macro (task planing) levels.

Robustness of 5G networks for tele-operated cooperation is evident as experimented in the wireless cooperative control of two platforms (one backhoe and one crawler dump truck, or one excavator and one front loader) [Reference Corporation88]. By using a 5G high-speed, high-capacity, and low-latency system, the onsite images of eight cameras with 8K resolutions including the sound have been streamed in real time over the distance up to 800 km to provide the operators with true feelings of the “on-field” condition, as illustrated in Fig. 5. The feasibility of wireless cooperative control of autonomous platforms has confirmed the possibility of IoT-based control framework under the integration of 5G network for RAS-based multiplatform coordination.

Figure 5.

Remote control in cooperative manner with 5G technology [Reference Corporation88].

For performance monitoring in autonomous operations of various earthmoving platforms, it is essential to accurately identify their activities on a site. To this end, vision-based systems are useful in extracting information from captured images, particularly when considering the interactive operations among equipment. An activity identification framework was proposed in ref. [Reference Kim, Chi and Seo89] to cover interactive aspects of earthmoving operations, including equipment tracking, action recognition of individual equipment, interaction analysis, and postprocessing. Typical interactions of excavators and dump trucks in earthmoving processed from thousands of image frames from actual construction sites could be used in analysis of automated operations of earthmoving platforms. Since the wireless information aggregated from the fusion of sensors, machine controls and network communications produce a huge amount of data, AI techniques such as the convolutional neural network (CNN) [Reference Kim, Chi and Seo89], or a deep learning algorithm [Reference Arabi, Haghighat and Sharma90] have been applied for processing, evaluating, validating, and predicting to eventually result in suitable control strategies for online cooperation.

Recent developments of machine learning and deep learning techniques have found many applications in construction automation such as equipment tracking or condition monitoring. As earthworks often involve cyclic operations, it is in favor for model training to learn repetitive features from collected data using vision-based systems. In ref. [Reference Kim and Chi91], a tracking-learning-detection algorithm is developed using a hybrid deep learning model with CNN and long short-term memory recurrent neural network (RNN) to automatically localize vehicles and recognize the sequential actions of robotic excavators such as digging, hauling, dumping, and swinging. These results are promising to be integrated into a cooperative framework to predict the next steps of interaction between heterogeneous platforms. To improve tracking accuracy, for example, in the nighttime or in poor lighting conditions, a deep learning illumination algorithm CNN-based feature extractor has been proposed in ref. [Reference Xiao, Lin and Chen92] for data assimilation and object identification in facing the challenge of illumination variations and motion blurs [Reference Xiao, Lin and Chen92]. Feature preservation capability of deep learning is also enhanced with a hierarchical CNN algorithm with autotuned thresholding for crack detection in built infrastructure [Reference Zhu, Dinh, Phung and Ha93]. Other applications of deep learning include also construction work management, sewer assessment, and 3D point cloud mapping [Reference Ma, Czerniawski and Leite94], or tele-robotic time delay mitigation [Reference Farajiparvar, Ying and Pandya38]. However, for training and applying effectively learning-based frameworks, underlying limitations, such as videos and images captured in different viewpoints, lighting conditions, multispectrum, and volatile weather, have to be resolved such as big data requirements for model training in general earthmoving conditions. Also, intensive computation for real-time processing and controlling high number of equipment, prior experiments for various building styles or locations should be take into account [Reference Khallaf and Khallaf95].

The teaming of multiplatforms on a site can be decentralized to leverage the performance of a cooperative policy in any scale of swarms via reinforcement learning [Reference Sartoretti, Wu, Paivine, Kumar, Koenig and Choset96]. Moreover, as teaming can be combined with other modes of control, interactions with human workers are also needed for the fleet cooperation in some scenarios, where a higher safety level should be ensured [Reference Frank, Ruvald, Johansson, Larsson and Larsson97]. Towards a new generation of intelligent earthmoving equipment, open and interoperable frameworks using AI have been developed for retrofitting to existing fleet of heavy construction machines [Reference Fretty98] to increase the equipment’s awareness to the environment and their ability to make decisions on coopervative tasks with other platforms in the team. These intelligent machines are aimed at acquiring the ability to “perceive their surroundings and be continuously alert, helping operators work more efficiently and safely” [99], as conceptually depicted in Fig. 6.

Figure 6.

Safe and efficient teaming of using embedded AI [99].

With disruptive technologies including CPS, 5G communication, and AI, teaming capacity in earthmoving automation is expected to be enhanced to achieve full autonomy on an earthmoving site. Embracing this vision, the Intracore Concept-X, has been launched as a versatile solution for autonomous earthworks with the ultimate aim of increasing productivity and safety, expected to be commercialized by 2025 [Reference Bennink43]. The work site topography will be surveyed via 3D drone scanning, whereupon 5G-based remote control technology will be deployed to operate RAS-based equipment such as excavators, wheel loaders, and haul trucks, as illustrated in Fig. 7.

Figure 7.

Teaming of autonomous machines in Concept-X [Reference Bennink43].