Introduction
In the recent years, the development of autonomous vehicles, especially autonomous cars, has been a common topic reported frequently in newspapers and television programs, thanks to the advance in control systems and the proliferation of machine learning techniques. Our research on collision avoidance for ships at sea aims to develop technologies that will eventually lead to the future autonomous ships that can not only steer in open waters and less congested areas but also are capable of avoiding collisions in congested harbors and reaching alongside berths without direct human involvement.
There have been various approaches to solving vehicle collision avoidance problems which can be divided into two large categories. One is vehicle control system development and the other traffic control system development, with the former being more relevant to this research. Vehicle control can be further categorized into the dynamical systems approach (e.g., Machado et al., Reference Machado, Malheiro, Monteiro, Erlhagen and Bicho2016), which relies on traditional control theories, and the intelligent systems approach (e.g., Yang et al., Reference Yang, Na and Heo2017), which employs knowledge-based systems and machine learning techniques to make collision avoidance decisions. Although the dynamical systems approach can be effectively applied in mostly predictable circumstances, when the uncertainty level becomes high and exceptions happen, the intelligent systems approach will be needed. Traditional knowledge-based systems have been applied to collision avoidance (e.g., Jin and Koyama, Reference Jin and Koyama1987). However, the issues of knowledge acquisition, formalization, and management have remained to be practically challenging.
The recent progress in machine learning, especially the deep learning (LeCun et al., Reference LeCun, Bengio and Hinton2015), has opened the ways to developing systems that can learn from humans’ operation experiences (e.g., through supervised deep learning) and from machines’ own experiences [e.g., through reinforcement learning (RL)]. The RL approach allows an agent to learn from its own experience. By interacting with the environment, the agent learns to select actions at any state to maximize its total reward. In case of deep learning, for example, AlphaGo (Chen, Reference Chen2016), the agent learns from the experience of human experts and apply the learned skills to solving the problems in the same domain of the experts. Furthermore, computer agents can learn to play games, for example, Go, by themselves through RL without any input from humans. AlphaGo-Zero is such an example that even won the human experience based AlphaGo in game matches (Silver et al., Reference Silver, Schrittwieser, Simonyan, Antonoglou, Huang, Guez, Hubert, Baker, Lai, Bolton, Chen, Lillicrap, Hui, Sifre, van den Driessche, Graepel and Hassabis2017).
In this research, the attempts have been made to apply the human experience-based supervised learning approach to train neural networks, but the effectiveness has been limited, largely because it was a nontrivial process to acquire sufficient human steering data to train the computer agents to a high skill level. Only rarely could we have opportunities to obtain meaningful data for complex ship encounters and the amount of such data is often too small for the agent training purpose. Therefore, we adopted the RL approach to acquire collision avoidance knowledge in complex ship encounter situations.
From a system point of view, the development of RL agents needs to satisfy two requirements. The first requirement relates to ship behavior design. Although through deep RL, ship agents can learn their optimal policies given the state and action spaces, the state transition, and reward functions, the learned neural network is still a black box that provides little identifiable knowledge about agents’ action mechanisms. To make agents’ behavior more transparent, it is important to understand how the agents’ behavior changes in response to the change of reward functions. In this paper, an initial effort in turning the reward functions for ship agent behavior design is presented.
Second, training in RL takes time, and the capability of transferring previously learned knowledge (i.e., neural network) in the new learning situations effectively is needed not only for speeding up the learning processes but also for expanding the range of situations by including the ones covered by other agents. This requirement calls for transfer learning in the reinforcement learning context. In this paper, a belief-based transfer RL method is introduced, and its design features are investigated and presented.
To summarize, in this research we address two research questions: (1) how the agent behavior is determined by reward functions in RL? and (2) what are the important parameters that impact both the learning and task behaviors of the agent in a transfer RL context?
The rest of the paper is organized as follows. The related work is reviewed in the section “Related work”, and then, the task of ship collision avoidance is described in the section “Tasks of ship collision avoidance”. “Machine learning based knowledge capture” provides details of a machine learning approach for collision avoidance, and section “Case study design” presents the design of case studies, and the results and discussion are described in the section “Results and discussion”. The conclusions are drawn in the section “Conclusions” together with the directions of future work.
Related work
Collision avoidance
Over the years, collision avoidance has been a common research topic in many industrial domains, including robotics, shipping, air traffic control, and self-driving cars. In the area of robotics, research has focused on issues related to how vehicle robots avoid obstacles and collisions with each other (Brunn, Reference Brunn1996; Alonso-Mora et al., Reference Alonso-Mora, Breitenmoser, Rufli, Beardsley, Siegwart and Martinoli2013; Shiomi et al., Reference Shiomi, Zanlungo, Hayashi and Kanda2014) and how assembly robots, or manipulators, avoid interferences among its own arms or with those of others (Hourtash et al., Reference Hourtash, Hingwe, Schena and Devengenzo2016; Hameed and Hasan, Reference Hameed and Hasan2016). In the sea and river shipping industry, collision avoidance can be highly difficult, when the water areas are becoming congested, due to the large inertia of ships causing immovability when movement is needed (Goerlandt and Kujala, Reference Goerlandt and Kujala2014). Once a collision happens at sea, the loss can be tremendous (Eleftheria et al., Reference Eleftheria, Apostolos and Markos2016). Airplane collision avoidance (Zou et al., Reference Zou, Alexander and McDermid2016) and even the collision with debris in space (Casanova et al., Reference Casanova, Tardioli and Lemaître2014) have become issues due to the increasing level of congestion. The self-driving industry is a rapidly growing field in the recent years, and fully autonomous cars are expected to become available soon. A self-driving system is considered to be more engineered in the sense that each car agent is equipped with more intelligence, which is at least able to (a) process human-level perceptions, that is, detecting lanes, traffic signs, pedestrians, other vehicles/obstacles, etc.; (b) predict trajectories of pedestrians and other vehicles; and (c) deal with complex social interactions with other vehicles and pedestrians. After all, self-driving cars must be able to avoid obstacles and other vehicles in various situations (Mukhtar et al., Reference Mukhtar, Xia and Tang2015; Kuderer et al., Reference Kuderer, Gulati and Burgard2015).
Collision avoidance problems have always attracted the attention of researchers in various research fields such as artificial intelligence, control theory, robotics, multi-agent system, and so on. The traditional practice to achieve real-time obstacle avoidance was to create an artificial potential field (Khatib, Reference Khatib1986). Fahimi et al. (Reference Fahimi, Nataraj and Ashrafiuon2009) proposed harmonic potential functions and the panel method to address multi-robot obstacle avoidance problem in the presence of both static and dynamic obstacles. Mastellone et al. (Reference Mastellone, Stipanovic, Graunke, Intlekofer and Spong2008) designed a controller for collision avoidance based on the Lyapunov-type approach and demonstrated the robustness of the system when the communication between robots was unreliable. Keller et al. (Reference Keller, Thakur, Gallier and Kumar2016) designed a path planner for unmanned aircraft systems to provide surveillance by combining graph search and B-spline parametric curve construction, which could successfully navigate around obstacles and provide sufficient coverage. Tang and Kumar (Reference Tang and Kumar2015) proposed the OMP + CHOP algorithm for a centralized multi-robot system, which was shown to be safe and complete, but at the cost of optimality.
Machine learning
In order for collision avoidance algorithms to be more adaptive and flexible in the real-world complex environment, learning capabilities of a multi-agent system have been developed. In the recent years, deep learning has achieved tremendous success in various areas such as image recognition (Krizhevsky et al., Reference Krizhevsky, Sutskever and Hinton2012; Le et al., Reference Le, Ranzato, Monga, Devin, Chen, Corrado, Dean and Ng2012), speech recognition (Hinton et al., Reference Hinton, Vinyals and Dean2015), automatic game playing (Mnih et al., Reference Mnih, Kavukcuoglu, Silver, Graves, Antonoglou, Wierstra and Riedmiller2013), self-driving (Bojarski et al., Reference Bojarski, Del Testa, Dworakowski, Firner, Flepp, Goyal, Jackel, Monfort, Muller, Zhang, Zhang, Zhao and Zieba2016; Ohn-bar and Trivedi, Reference Ohn-Bar and Trivedi2016), and so on. Deep learning algorithms can extract high-level features by utilizing deep neural networks, such as convolutional neural networks (CNNs) (Krizhevsky et al., Reference Krizhevsky, Sutskever and Hinton2012), multi-layer perceptrons, and recurrent neural networks (RNNs) (LeCun et al., Reference LeCun, Bengio and Hinton2015). Scaling up deep learning algorithms is able to discover high-level features in a complex task. Dean et al. (Reference Dean, Corrado, Monga, Kai, Devin, Mao, Ranzato, Senior, Tucker, Yang, Le and Ng2012) constructed a very large system, which was able to train 1 billion parameters using 16,000 CPU cores. Coates et al. (Reference Coates, Huval, Wang, Wu and Ng2013) scaled to networks with over 11 billion parameters using a cluster of GPU servers.
RL is about an agent interacting with the environment, learning an optimal-policy, by trial-and-error, for sequential decision-making problems in a wide range of fields in both natural and social sciences, and engineering (Sutton and Barto, Reference Sutton and Barto2018; Bertsekas and Tsitsiklis, Reference Bertsekas and Tsitsiklis1996; Matarić, Reference Matarić, Arkin and Bekey1997; Szepesvari, Reference Szepesvari2010). Mnih et al. (Reference Mnih, Kavukcuoglu, Silver, Graves, Antonoglou, Wierstra and Riedmiller2013) introduced a deep learning algorithm using experience replay and CNNs to learn a Q-function, which can play various Atari 2600 games better than human players. Experience replay allows an online learning agent to random sample batches from past experiences to update Q-values, thus breaking the correlations between consecutive frames. By combining the supervised learning and RL approach, the group at DeepMind has further proven that their deep learning algorithm can outperform a world champion in the most challenging classic game Go (Silver et al., Reference Silver, Huang, Maddison, Guez, Sifre, van den Driessche, Schrittwieser, Antonoglou, Panneershelvam, Lanctot, Dieleman, Grewe, Nham, Kalchbrenner, Sutskever, Lillicrap, Leach, Kavukcuoglu, Graepel and Hassabis2016, Reference Silver, Schrittwieser, Simonyan, Antonoglou, Huang, Guez, Hubert, Baker, Lai, Bolton, Chen, Lillicrap, Hui, Sifre, van den Driessche, Graepel and Hassabis2017; Churchland and Sejnowski, Reference Churchland and Sejnowski2016; Wang et al., Reference Wang, Zhang, Zheng, Wang, Yuan, Dai, Zhang and Yang2016a), which has extremely large number of possible configurations, and is difficult to evaluate board positions. Schaul et al. (Reference Schaul, Quan, Antonoglou and Silver2016) further developed a prioritized experience replay framework to sample more important transitions and learn more efficiently. For reward function studies, there have been ways to do inverse RL (Ng and Russell, Reference Ng and Russell2000), reward strategy analysis (Manju and Punithavalli, 2011), and reward function generation (Mericli et al., Reference Mericli, Mericli and Akin2010).
Chen et al. (Reference Chen, Liu, Everett and How2016) developed a decentralized multi-agent collision avoidance algorithm based on deep RL. Two agents were simulated to navigate toward their own goal positions and learn a value network which encodes the expected time to goal, and the solution was then generalized in multi-agent scenarios. Deep learning algorithms have been successful in achieving end-to-end learning. Dieleman and Schrauwen (Reference Dieleman and Schrauwen2014) investigated whether it is possible to apply feature learning directly to raw audio signals by training CNNs. Traditionally, content-based music information retrieval tasks are resolved based on engineered features and shallow processing architectures, which relies on mid-level representations of music audio, for example, spectrograms. The results showed that even though the end-to-end learning does not outperform the spectrogram-based approach, the system is able to learn automatically frequency decompositions and feature representations from raw audio.
Self-driving cars mentioned above is a promising field which took off in the last few years and heavily relies on the advances in deep learning. Since self-driving cars always require a great deal of expensive and complex hardware, Yu et al. (Reference Yu, Palefsky-Smith and Bedi2016) implemented a deep Q-learning algorithm using dataset (images) from real-time play of the game JavaScript Racer. In a recent published paper (Bojarski et al., Reference Bojarski, Del Testa, Dworakowski, Firner, Flepp, Goyal, Jackel, Monfort, Muller, Zhang, Zhang, Zhao and Zieba2016), a CNN is trained to map steering commands directly from raw pixels from camera input. The system automatically learned internal processing steps such as detecting useful road features with only the human steering angle as the training signal. This end-to-end learning approach is challenging in that it requires a huge number of inputs and the advantage is that it releases the rely on the designer's prior domain knowledge.
Transfer learning
Given a complicated task which is difficult to learn directly, transfer learning is a commonly used technique which can generalize previously learned experience and apply these experiences into new tasks (Arnold et al., Reference Arnold, Nallapati and Cohen2007; Pan and Yang, Reference Pan and Yang2010; Bahadori et al., Reference Bahadori, Liu and Zhang2014). Transfer learning refers to utilizing knowledge gained from source tasks to solve a target task. It is believed that in an RL context, transfer learning can speed up the learning agent to learn a new but related task (i.e., target task) by learning source tasks first. Taylor and Stone (Reference Taylor and Stone2007) introduced a transfer algorithm called Rule Transfer, which summarizes source task policy, modifies the decision list, and generates a policy for the target task. Rule learning is well understood and human readable. The agent benefits from the decision list initially and continues to refine its policy through target task training. It was shown that Rule Transfer could significantly improve learning in robot soccer using learned policy from a grid-world task.
Fernandez and Veloso (Reference Fernandez and Veloso2006) proposed two algorithms to address the challenges of Policy Reuse in an RL agent. The major components include an exploration strategy and a similarity function to estimate the similarity between past policies and new ones. The PRQ-learning algorithm probabilistically bias an exploration learning process by using a Policy Library. In the second algorithm called PLPR, the Policy Library is created when learning new policies and reusing past policies.
Torrey et al. (Reference Torrey, Shavlik, Walker and Maclin2006) introduced the induction logic programming for analyzing the previous experience of source task and transferred rules for when to take actions. Through an advice-taking algorithm, the target task learner could benefit from outside imperfect guidance. A system AI 2 (Advice via Induction and Instruction) for transfer learning in RL was built, which creates relational transfer advice using inductive logic programming. Based on a human-provided mapping from source tasks to target tasks, the system was able to speed up RL.
In transfer learning within deep neural networks, a base network on a base dataset and task is first trained, and the learned features are then transferred to the target network to be trained on a target dataset and task. Transfer learning can be particularly useful to train a large target network without overfitting if the target dataset is much smaller than the base/source dataset. One common approach is to copy the first n layers of the base network to the first n layers of the target network, while the remaining layers of the target network are randomly initialized and trained. Yosinski et al. (Reference Yosinski, Clune, Bengio and Lipson2014) presented a way to measure the degree to which a certain layer is general or specific and found that initializing a network with transferred features from almost any layers could boost the performance after fine-tuning to a new dataset. A task-driven deep transfer learning framework for image classification was designed (Ding et al., Reference Ding, Nasrabadi and Fu2016), where the features and classifiers are obtained at the same time. Through pseudo labels for target domain, the system could transfer more discriminative information to the target domain. Parisotto et al. (Reference Parisotto, Ba and Salakhutdinov2016) proposed a transfer reinforcement learning approach (Actor-Mimic) to mimic expert decisions for multi-task learning, which adopts the concept of policy distillation (Hinton et al., Reference Hinton, Vinyals and Dean2015).
To date, there has been little literature aiming to combine deep RL and transfer learning to solve robotic collision avoidance problems, because (a) it is difficult to directly learn from raw pixel or distance sensory inputs and (b) it requires large amount of training data, which is not easy to generate in real life. This research combines transfer and RL approaches for ship agents to acquire collision avoidance knowledge more efficiently.
Tasks of ship collision avoidance
Collision avoidance at sea
Collision avoidance at sea involves a ship agent, called own ship, a number of target ships and obstacles, and the designated routes for different types (e.g., different size) of ships. Figure 1a illustrates a typical ship encounter situation, where Ship-A is the own ship, Ships B, C, D, and E are target ships of the own ship. Each ship has its own destinations such as a specific berth when moving into a port or an exit direction when moving out. There are also obstacles, such as Obs-a, Obs-b, and Obs-c, which can be islands, fixtures, or buoys. The routes are often indicated on the hardcopy or electronic maps, called charts. In addition, there are regulations, dictated by COLREG rules (Ford, Reference Ford2009), by which the ships will be considered as having the “right of the road” or should “give way” to other ships depending on their position in the encounter situation. Although the regulations are supposed to be followed by all the ships, there is no guarantee that a target ship will follow the rules, especially when the situation can be interpreted in different ways by different ships.
Ship collision avoidance at sea.
In this research, since our focus is to investigate the issues related to RL-based collision avoidance knowledge capture, we define a simplified version of the tasks of collision avoidance. As shown in Figure 1b, the own ship is on track moving north (or upward). The destination of the own ship is any point on the upper edge of the rectangle, indicated as “Goal”. So the “success” of the own ship in this task is defined as reaching the goal without hitting any other ships or obstacles, and the “efficiency” of the own ship in a successful run is defined by the normalized inverse of the total number of units of distance (or steps) traveled by the ship. The efficiency of unsuccessful runs is defined as 0 (zero). There can be stationary obstacles and moving ships in the water area. The objective of RL based knowledge capture is to train ship agents, so that they can achieve successful runs with high efficiency for various encounter situations.
Task complexity and similarity
The collision avoidance tasks indicated in Figure 1 can be simple or more complex depending on how many target ships and obstacles are involved and how difficult the encounter situation can be. From a knowledge capture perspective, it will be ideal that the ship agents can be trained in simple situations and the knowledge (i.e., neural networks) obtained from the simpler situations can be applied and extended through further learning in more complex situations. It is conceivable that knowledge transfer between similar cases tends to be easier than that between dissimilar cases. To facilitate the knowledge transfer study, simple measures of task complexity and similarity are introduced as follows.
The complexity of a collision avoidance task can be generally considered to have two major components, namely internal complexity and external complexity. The internal complexity stems from how difficult the own ship can be maneuvered to avoid collision (e.g., large ships are hard to move, and smaller ones easier to move), while the external complexity captures how difficult the encounter situation can be to avoid collision. Since moving targets are harder to avoid than stationary obstacles and more targets or obstacles are harder to deal with, the external complexity can be further decomposed into object complexity – measured by whether an object is moving (ship) or stationary (obstacle) and aggregate complexity – measured by the total number of target ships and obstacles. Figure 2 illustrates such a complexity hierarchy.
Collision avoidance task complexity hierarchy.
In this research, the similarity between two collision avoidance tasks is measured by the inverse of the “distance” between the two tasks in the three-dimensional task complexity space, as indicated in Figure 3a. The three axes in the figure are x-axis is the obstacle dynamics plane (red), y-axis is the number of obstacles plane (green), and z-axis is the vehicle dynamics plane (blue). Figure 3b shows a specific contextualization of this task complexity model, with the three dimensions having specific value ranges specified as Vehicle dynamics = {large ship, small ship}, Obstacle dynamics = {static, moving}, and Number of obstacles = {single, multiple}.
Complexity spaces and task locations.
For any two tasks i and j in Figure 3b, their task complexity can be determined by two vectors: 
$\overrightarrow {{\rm complexity}} _i = \lpar {{\rm O}{\rm D}_i\comma \,{\rm N}{\rm O}_i \comma \,{\rm V}{\rm D}_i} \rpar $ and 
$\overrightarrow {{\rm complexity}} _j = ({\rm O}{\rm D}_j\comma \,{\rm N}{\rm O}_j\comma \,{\rm V}{\rm D}_j)$, respectively, where OD is obstacle dynamics, NO is the number of obstacles, VD is vehicle dynamics. The weighted length of the vectors from the origin can be used as a scaler measure for the task complexity, that is,
$$|| \overrightarrow {{\rm complexity}} _i || = \sqrt {\lambda _1{\rm OD}_i^2 + \lambda _2{\rm NO}_i^2 + \lambda _3{\rm VD}_i^2}. $$In this research, the inter-task similarity of the two tasks, i and j is defined as the distance between the two points in the complexity space given as follows:
$${\rm similarity}_{i\comma j} = \displaystyle{1 \over {1 + \sqrt {{\lpar {{\rm O}{\rm D}_i-{\rm O}{\rm D}_j} \rpar }^2 + {\lpar {{\rm N}{\rm O}_i-{\rm N}{\rm O}_j} \rpar }^2 + {\lpar {{\rm V}{\rm D}_i-{\rm V}{\rm D}_j} \rpar }^2}}}. $$In general, the three factors may not be equally important in determining the inter-task similarity. Thus, three additional weights can be added, as we did in Eq (1):
$$\eqalign{&{\rm similarity}_{i\comma j} = \cr &\ \ \displaystyle{1 \over {1 + \sqrt {\lambda _1{\lpar {{\rm O}{\rm D}_i-{\rm O}{\rm D}_j} \rpar }^2 + \lambda _2{\lpar {{\rm N}{\rm O}_i-{\rm N}{\rm O}_j} \rpar }^2 + {\rm \lambda} _3{\lpar {{\rm V}{\rm D}_i-{\rm V}{\rm D}_j} \rpar }^2}}}}. $$Determining the values of the relative weights of λ 1, λ 2, and λ 3 require a deep understanding of the collision avoidance tasks in corresponding domains. In the case study section of this paper, a simplified measuring scheme is adopted for simulation and training purposes.
Machine learning based knowledge capture
For the collision avoidance tasks and the similarity definition introduced above, an RL approach is adopted to (1) capture the collision avoidance knowledge through trial-and-error and (2) transfer the captured knowledge to closely related but more complex tasks. Following subsections describe the principles and algorithms of the methods introduced.
Knowledge capture: deep RL
RL for collision avoidance involves sensory data and collection of sufficient data through trial-and-error processes. However, it can be difficult to collect a sufficient amount of data as the training input by only relying on sensory inputs. In addition, the state/action space is always continuous which makes it impractical to build a look-up Q-table. To overcome the curse of dimensionality, deep neural networks are used as functional approximators to replace the Q-table and approximate Q-values.
We began this research by implementing the deep RL algorithm with experience replay as proposed in Mnih et al. (Reference Mnih, Kavukcuoglu, Silver, Graves, Antonoglou, Wierstra and Riedmiller2013). First, standard Q-learning (Watkins, Reference Watkins1989) is considered, which can be formulated as a tuple of
$${\rm Q{\hbox-}learning}:\left\langle {S\comma \,A\comma \,P\comma \,R\comma \,\gamma} \right\rangle, $$where S = {s 1, … }: state space; A = {a 1, …} : action space; P : transition matrix; 
$R_t = \sum\nolimits_{{t}^{\prime}}^T {\gamma ^{{t}^{\prime}-t}r_{{t}^{\prime}}}$ : reward function; and γ : discount rate.
In Eq. (4), S is the state space, which consists of agent's all possible states in the environment. A is the action space consisting of all the possible actions that the agent can take. P is the transition matrix (usually unknown in a model-free environment), R is the reward function, and γ is the discount factor. At any given time t, the agent's goal is to maximize its future discounted return 
$R_t = \sum\nolimits_{{t}^{\prime}}^T {\gamma ^{{t}^{\prime}-t}r_{{t}^{\prime}}}$, where T is the time when the game or episode terminates. Like many other RL algorithms, the agent estimates at each time step the action value function Q(s, a), using a Bellman equation, Eq. (5), as an update. Such value iteration algorithms converge to the optimal value function:
$$Q_{i + 1}(s\comma \,a) = {\bf {\mathbb{E}}}\left[ {r + \gamma \mathop {\max} \limits_{{a}^{\prime}} Q_i({s}^{\prime}\comma \,{a}^{\prime})\vert {s\comma \,a} } \right].$$In order to adapt to tasks involving infinitely large state/action space where building the Q-table is impractical, deep Q-learning with experience replay uses a neural network as a function approximator (Q-network). A Q-network with weights θ i can be trained by minimizing the loss function L i(θ i) at each iteration i,
$$L_i(\theta _i) = {\bf {\mathbb{E}}}\lsqb {(y_i-Q{(s\comma \,a;\theta_i)}^2} \rsqb, $$where 
$y_i = {\bf {\mathbb E}}\big[ {r + \gamma \mathop {\max} \limits_{{a}^{\prime}} Q({s}^{\prime}\comma \,{a}^{\prime};\theta_{i-1})} \big]$ is the target Q-value for iteration i. The gradient is calculated by the following:
$$\eqalign{&\nabla _{\theta _i}L_i(\theta _i) = \cr &\ \ \ \ {\bf {\mathbb E}}_{s\comma \,a\comma \,r\comma \,{s}^{\prime}}\left[ {\left( {r + \gamma \mathop {\max} \limits_{{a}^{\prime}} Q({s}^{\prime}\comma \,{a}^{\prime};\theta_{i-1})-Q(s\comma \,a;\theta_i)} \right)\nabla_{\theta_i}Q(s\comma \,a;\theta_i)} \right].}$$The deep Q-learning algorithm utilizes a technique called experience replay where the agents’ experiences, e t = (s t, a t, r t, s t+1), are stored into a replay memory, D = {e 1, e 2, …, e N} (N is the capacity of the replay memory). Then, mini-batches are randomly sampled from D and applied to Q-learning updates. The agent selects an action according to the ɛ-greedy policy.
Various approaches have been proposed to stabilize learning process such as deep Q-network (DQN) (Mnih et al., Reference Mnih, Kavukcuoglu, Silver, Graves, Antonoglou, Wierstra and Riedmiller2013), double DQN (van Hasselt et al., Reference van Hasselt, Guez and Silver2015), and dueling DQN (Wang et al., Reference Wang, School, Hessel, van Haselt, Lanctot and de Freitas2016b). In this research, our base network is built by combining these three approaches. The double DQN algorithm is able to solve the problem of overoptimistic value estimates, by separating the target network, which is used for action evaluation, from the current network which is used for action selection. The agent's experience is randomly assigned to update one of the two networks.
In standard DQN, at each update of the Q-values, only the value for one of the actions is updated, whereas others remain untouched. Dueling DQN separates the Q-value into a state value and action advantages, so that the state value is updated more frequently.
Agent learning behavior
In this research, a computer game environment was created to conduct case studies, as shown in Figure 4. The game environment consists of a learning agent (the green ball), static and moving obstacles (the red balls), and a goal area (the orange rectangle). The dark slice on the ball indicates the direction of the ball's movement. An end-to-end deep learning approach (Mnih et al., Reference Mnih, Kavukcuoglu, Silver, Graves, Antonoglou, Wierstra and Riedmiller2013) is taken and the game window images are captured as input state information. Therefore, a state s i ∈ S in Eq. (4) for this study is defined as the pixel values of the game window. Figure 4 shows an example of the game window.
A screen shoot of the game system for case studies.
The action space is composed of seven actions, A = {a 1, …, a 7}, as indicated in Table 1. Each action is defined by a pair of forward velocity v, and angular velocity ω, that is, a i = (v i, ω i), i = 1, 2, …, 7.
Agent actions
Reward functions and knowledge capture
For tasks like ship collision avoidance, usually the reward function is composed of two major items. One is ending reward, or terminal reward, that rewards the agent when the game or episode is completed (hence ending or terminal). Typically, a large positive reword is given for success and a large negative reward for failure. The other reward item is shaping reward, which signifies how the agent will be rewarded during the process of the task execution. Typically, a small negative reward is given for every step of action or every unit time because staying in the game (or process) should be penalized, so that the agent will attempt to complete the task as soon as possible. Based on this consideration, the following Typical Reward Function is applied as a baseline design:
$$r^t = \left\{ {\matrix{ {200\comma \,\;} \hfill & {{\rm if\;} \,{\rm reach\;} \,{\rm goal}} \hfill \cr {-200\comma \,} \hfill & {{\rm if}\,{\rm \; hit}\,{\rm \; obstacle}} \hfill \cr {-1\comma \,\;} \hfill & {{\rm else\;} \;} \hfill \cr}} \right. . $$Since one of the research questions mentioned above relates to assessing how variations of reward function can influence the agents’ learning and performing behavior, we need to introduce untypical or variable reward functions. Because the agents’ task performing behavior occurs during the process of task execution, rather than at the ending point of the process, the focus of varying reward functions can be put on the shaping reward item. More specifically, we attempt to shape the agents’ behavior so that the efficiency of collision avoidance, as defined in the section “Collision avoidance at sea”, can be increased.
Figure 5 illustrates an example of the efficiency difference between different task performing behaviors of a ship agent. Given the obstacles in the center area, the agent can go either the two side paths, A and C, or the center path, B. Although all three paths can lead to a successful run, the efficiency of path B is much better since it is the shortest path for reaching the goal.
Agent task performing behavior in collision avoidance.
In order for the agent to learn explicitly the most efficient behavior, for every step from start to end, the agent receives an additional reward, which is computed by a goal reward r g and a deviation reward r dev. Thus, at any time step t, the reward function is defined as the following Enhanced Reward Function:
$$r^t = \left\{ {\matrix{ {200\comma \,} \hfill & {{\rm if} \,{\rm reach}\,{\rm goal}} \hfill \cr {-200\comma \,\;} \hfill & {{\rm if} \,{\rm hit}\,{\rm obstacle}} \hfill \cr {-(\omega_{\rm g}\cdot r_{\rm g} + \omega_{{\rm dev}}\cdot r_{{\rm dev}})\comma \,} \hfill & {{\rm else\;}} \hfill \cr}} \right. .$$The goal reward, r g, is the reward of moving toward goal direction, and the deviation reward, r dev, is the negative reward of deviating from the goal direction. Currently, r g and r dev are computed by a linear function (10) of agent's current location coordinates, y and x dev, where x dev is the agent's deviation from the horizontal center of the window: 
$x_{\rm dev} = {\rm abs}\lpar {x/{\rm width}-1/2} \rpar$, and width is the horizontal length of the game window, k g, b g, k dev and b dev are constant coefficients. To make the learning more stable, r g and r dev are clipped between [−1, 1] and [0, 0.5], respectively. ω g and ω dev are the relative weight on r g and r dev:
$$\left\{ {\matrix{ {r_{\rm g} = k_{\rm g}y + b_{\rm g}} \hfill & {\in \lsqb {-1\comma \,1} \rsqb } \hfill \cr {r_{{\rm dev}} = k_{{\rm dev}}x_{{\rm dev}} + b_{{\rm dev}}} \hfill & {\; \in \lsqb {0\comma \, 0.5} \rsqb } \hfill \cr}} \right. .$$It is worth mentioning that the shaping reward designed this way can be used to “design” the agent's behaviors as will be discussed later.
Knowledge transfer: a belief-based approach
As mentioned above, RL has been successfully applied to capture domain knowledge for various tasks including the game of Go (Silver et al., Reference Silver, Huang, Maddison, Guez, Sifre, van den Driessche, Schrittwieser, Antonoglou, Panneershelvam, Lanctot, Dieleman, Grewe, Nham, Kalchbrenner, Sutskever, Lillicrap, Leach, Kavukcuoglu, Graepel and Hassabis2016, Reference Silver, Schrittwieser, Simonyan, Antonoglou, Huang, Guez, Hubert, Baker, Lai, Bolton, Chen, Lillicrap, Hui, Sifre, van den Driessche, Graepel and Hassabis2017). However, RL can be expensive in terms of the time for training and that for exploring various possible situations. Furthermore, for highly complex task situations, RL may not be able to converge at the desired level of total reward. To deal with these issues, transfer RL has been explored to expand the reach of RL.
Expert agent and student agent
In this research, we consider that two agents are involved in transfer RL, an expert agent and a student agent. The expert agent is first trained through ordinary RL with a relatively simple task, called source task, and becomes an “expert” in that task. The knowledge (i.e., the neural network) of the expert can then be transferred to the student agent that is completely know-nothing at the beginning but can be “taught” by the expert agent through (1) receiving part or all of the expert's knowledge as initialization so that the student holds part or all of the knowledge for the source task and (2) asking the expert agent for help from time to time when the student agent ventures to learning closely related but more complex new tasks, called target tasks. “What knowledge and how much” should be passed from the expert to the student, and “what help and how often” should be provided by the expert to the student are two important questions.
Belief-based transfer learning
The goal of transfer learning is to transfer the expert's knowledge to the student agent, so that the student can applied the expert's knowledge to engage in learning with the closely related but more complex tasks more effectively – that is, converge to a high level of total reward – and efficiently – that is, fast ramping-up and fast convergence. In this research, we consider three phases of transfer learning. First is “copy expert” knowledge (neural network) as the student's initial neural network, as shown in Figure 6. If the student agent plans to work only with the same tasks as the expert did, then “copy expert” alone will be sufficient. However, the student agent wants to engage in learning in the closely related but more complex new target tasks. Therefore, the second phase “consult expert” is needed in which the student occasionally asks the expert for suggested actions for given states. As the student's learning progresses, the expert's help decreases until being phased out completely, leading to the last phase, called “learn by self”, where the student performs ordinary RL without further influence from the expert. Figure 6 shows an illustration of the three phases of the transfer RL together of the starting and ending points.
Three phases of transfer RL.
As shown in Figure 6, “consult expert” is the phase where the expert's “helping” or “coaching” happens. In this phase, we differentiate between three types of student actions based on the mechanism that produces the actions for the given state information.
a) Transfer action: With probability p 1 in Eq. (11a), the student agent seeks the help from the expert and selects the transfer action which is produced by directly using the expert neural network (see Fig. 6). T trans is a hyperparameter, called transfer period, during which the student is “helped” by the expert, and the frequency of help decreases linearly from β 0 at the beginning to zero at T trans. Thus, the transfer period T trans equals the duration of the “consult expert” phase in Figure 6. β 0 is another hyperparameter, called initial transfer belief, which indicates how much confidence the student puts in the expert knowledge. The higher the confidence, the more often the student tends to ask the expert for help.
(11a)
$$p_1 = \left\{ {\matrix{ {\beta_0\left( {1-\displaystyle{t \over {T_{{\rm trans}}}}} \right)\comma \,} \hfill & {\; t \le T_{{\rm trans}}} \hfill \cr {0\comma \,} \hfill & {t \gt T_{{\rm trans}}} \hfill \cr}} \right.$$(11b)$$p_2 = \varepsilon \lpar {1-p_1} \rpar $$(11c)$$p_3 = \lpar {1-\varepsilon} \rpar \lpar {1-p_1} \rpar $$(11d)$$p_1 + p_2 + p_3 = 1$$b) Exploration action: With probability p 2 of Eq. (11b), the student agent selects a random action. In this research, an ɛ-greedy method is adopted for ordinary RL, in which the exploration probability usually decreases linearly from 1 to a fixed small number, say 0.01, over a period of time T expl, T expl ≥ T trans. In this research, in order to leave rooms for “transfer actions”, the decreasing of the ɛ-value adopts a nonlinear concave shape rather than a linear shape. Therefore, the exploration probability decreases nonlinearly splitting a large portion from “exploitation” until T trans. Our experiment results have shown that without such a large portion of exploration, the student agent will not learn about the new task effectively. The reason is partly that the initial student network is the same as the transfer network, so the “exploitation” is the same as, or similar to “transfer”. Too little exploration will cause insufficient expansion to try the new tasks, leading to ineffective learning.
c) Exploitation action: With probability p 3 of Eq. (11c), the student agent selects the current best action produced by its own learned neural network, as in the ordinary RL. It should be noted that the expert network (the square NN in Fig. 6) is not updated during the student training process. Only the student network (the trapezoid NN in Fig. 6) is updated in the target task domain. As the learning converges and transfer period T trans passes, the expert network will be completely phased out.
Case study design
As indicated in Figure 6, there are different learning paths for a student agent to move from the “start” to the “end”. First, the student can learn from scratch by moving from “start” directly to phase 3 “learn by self”. Second, the student can select phase 1 “copy expert” and then move to phase 3 “learn by self”. Lastly, the student agent can take the whole course of “copy expert”, “consult expert”, and then “learn by self”. In the following, these three leaning paths are called L-Path1, L-Path2, and L-Path3, respectively.
Table 2 categorizes the case studies that are performed in this research including the cases, the parameters, and used parameter values. The reward function exploration is carried out only for L-Path1, since mixing the knowledge transfer may confound the effect of reward function. Both L-Path2 and L-Path3 are explored to assess the impact of pure “copy expert” and the inclusion of “consult expert”. The cases shown in Table 2 are carried out against various tasks and task transfer situations, which are discussed in detail in the following section “Tasks for training and transfer”.
Cases, parameters, and their values
Training parameters
As mentioned in the section “Agent learning behavior”, an end-to-end RL approach (Mnih et al., Reference Mnih, Kavukcuoglu, Silver, Graves, Antonoglou, Wierstra and Riedmiller2013) is employed in this research. The game window images, such as the one shown in Figure 4, are captured as input state information. The size of the game window is 400 × 400 pixels. During preprocessing, the window is first scaled down to 84 × 84 pixels, and then converted to grayscale since the color should not influence the agent's decision-making. At each step, the agent stacks 4 previous frames together, which are then fed into the neural network. The network structure is the same as the original DQN paper (Mnih et al., Reference Mnih, Kavukcuoglu, Silver, Graves, Antonoglou, Wierstra and Riedmiller2013) with 84 × 84 pixels input and an output of 7 actions. The experience replay size is 50,000. At every step, a mini-batch of size 32 is randomly sampled from the experience replay.
All the case studies were trained using Adam optimizer (Kingma and Ba, Reference Kingma and Ba2015) with a learning rate of 0.0001. The discount factor γ is set to be 0.99. The agent follows ε-greedy policy, with ε being annealed down from 1 to 0.01 during the first 1 million frames. The weight on goal reward ω g is 1. The weight on deviation reward ω dev can be 0, 0.5, 1.0, 1.5, or 2.0 for this study. Table 3 shows major system parameters and their values of the case studies.
System training parameters
Tasks for training and transfer
In this case study, four different tasks are considered, as shown in Figure 7. Task A is a simple task, in which there is only one static obstacle in a small restricted area. The position of the obstacle is randomly set within the restricted area for each episode during training.
Collision avoidance Tasks A through D used for case studies.
Task B is more complex than Task A in that there are two static obstacles randomly generated for each episode within a larger area. Task C and Task D involve moving obstacles (i.e., target ships) with the constant moving speed at 20 pixels per time step. Again, the initial position and moving direction of the target ships in Task C and Task D are set randomly for each episode.
Task B is used for reward function study, in which both typical reward function and enhanced reward function are tried, and their training results compared.
For the knowledge transfer studies, Task A is used as the source task, and the rest of the tasks are used as target tasks. To assess the complexity of each task and the similarity level between the tasks, we make the following assumptions.
1. The own ship is a small ship with desirable movability. Hence, for all tasks, the vehicle dynamics VD = 1.0.
2. The number of obstacles and/or target ships is either 1 (NO = 1.0) or 2 (NO = 2.0) depending on how many of them are involved.
3. For stationary obstacles, the object dynamics is 1 (OD = 1.0); for moving target ships, it is 2 (OD = 2.0).
4. Given that moving target ships are much more difficult to avoid and the own ship vehicle dynamics is minimum (for small ships), we set up the weight coefficients as follows: λ 1 = 3, λ 2 = 1.5, and λ 3 = 1.
Based on the above, we have the complexity vector for each task in Figure 7 represented as follows: 
$\overrightarrow {{\rm complexity}} _{\rm A} = \lpar {1.0\comma \, 1.0\comma \, 1.0} \rpar$, 
$\overrightarrow {{\rm complexity}} _{\rm B} = \lpar {1.0\comma \, 2.0\comma \,1.0} \rpar$, 
$\overrightarrow {{\rm complexity}} _{\rm C} = \lpar {2.0\comma \, 1.0\comma \, 1.0} \rpar$, and 
$\overrightarrow {{\rm complexity}} _{\rm D} = \lpar {2.0\comma \, 2.0\comma \, 2.0} \rpar$. Based on these vectors and the Eqs (1) and (3), the complexity values of each task and the similarity values between the tasks can be calculated, as shown in Table 4.
Task complexities and similarities
Results and discussion
In this research, four case studies are performed to investigate the RL based knowledge capture. They are reward function for behavior design, copy expert for knowledge transfer, adjust transfer period, and adjust transfer belief for knowledge transfer. In the following, we first discuss the results of reward function study and then provide details of how knowledge transfer strategies interact with the tasks of different similarities.
Reward functions for designing system behavior
In this study, Task B in Figure 7 is used as the learning task and L-Path2 is used for agent training. The ship agent starts from scratch and captures collision avoidance knowledge through ordinary RL (i.e., there is no knowledge transfer involved). The goal of this study is to explore the effect of applying the enhanced reward function of Eqs (9) and (10) and investigate the behavioral response of the ship agent to the changing reward function variables. The typical reward function [Eq. (8)] is used as a baseline case. Training of ship agents is carried out with varying values of the reward function parameter: deviation weight ω dev. Five values of ω dev were tried: 0, 0.5, 1.0, 1.5, and 2.0. The agent's learning speed and the rate of the successful runs (see “Collision avoidance at sea” section) are used as the performance measurements.
Figure 8 shows the average reward of each case. The x-axis is the number of episodes into the training process, and the y-axis is the average reward. The different colored lines correspond to different deviation weights. The deviation weight ω dev = 0, that is, Eq. (8) is used as reward function, is treated as the baseline case.
Learning performance – average reward for different ω dev values.
From Figure 8, it can be seen that adding a small (negative) deviation reward can boost learning speed. The reward plots of ω dev = 0.5, 1.0, and 1.5 (orange, green, and red curves, respectively) ramp up much faster than the baseline. At 100 × 100 episodes, it is already more than three times (3×) better, and at 150 × 100, about one and half times (1.5×) better.
Since the shaping rewards in these cases are all different, it is unreasonable to compare the agent's average reward after the convergence. In fact, they all converge to a similar range of reward (100–150).
To further evaluate the agent's learned behavior in different cases, 500 random tests were carried out for each ω dev setting, where the success rate and the average time of reaching the goal were recorded.
Figure 9 summarizes the average time of reaching the goal and the success rate of different cases with varying deviation weights. The baseline uses the typical reward function. All the statistics of average goal reaching time have been converted to relative percentage of the baseline. As can be seen, after adding deviation reward (penalty) to the reward function, the agent saves 5%, 14%, 11%, and 11% of time, as well as energy, in cases where ω dev = 0.5, 1.0, 1.5, and 2.0, respectively. When ω dev = 1.0 the goal reaching time is the best, only 86% of the baseline time. However, in all these cases, the success rate drops slightly. When ω dev keeps increasing to 1.5 or 2.0, the success rate drops to 0.91 and 0.89.
Task performance – goal reach time and success rate for different ω dev values.
The simulation results of the baseline case (typical reward function) and various reward tuning cases (enhanced reward function with varying ω dev) have demonstrated the impact of reward functions on the agent's learning and working behavior. Such an impact can be very sensitive to small changes in the values of the reward function parameters. Following are some insights drawn from the results.
Ending reward and interim reward: The baseline is based on the typical reward function in Eq. (8). The major feature is that it has two important step-function like “ending-rewards”, one for reaching the goal +200, and the other for hitting an obstacle −200, signifying two opposite endings of the episode. The “−1” in (8) and the added − (ω g · r g + ω dev · r dev) in (9) are “interim awards” that are devised to guide the way for the agent to reach its ending point. A close look at the work behavior of the agent trained based on Eq. (8) reveals that the agent always tries to avoid the whole “congested” water area and takes the side ways to reach the goal no matter how the obstacles in the area are positioned, as shown as Path A and Path C in Figure 5. By adding the penalty for horizontal deviation ω dev · r dev the agent's behavior changed: it seeks both the middle opportunities, for example, Path B, as well as the safer sideways, Path A and Path C. The added negative reward encouraged the agent to explore more the middle ways despite the collision risks. The result is that the average goal reaching time is shortened, but the success rate suffers a bit too. The interim reward provides an effective means to “design” the agent's behavior and embedding the rules, regulations and heuristic knowledge.
Sparse reward versus shaped reward: The reward functions with only “ending reward” items are sparse. Although the baseline function Eq. (8) had “−1” as its interim reward, it is sparser compared to function (9) which has more interim items involved. Sparse rewards slow down learning because the agent needs to take many actions before getting any significant reward. To avoid sparse rewards, the interim reward items are used to create shaped reward function (hence also called shaping rewards). As shown in Figure 8, shaped reward function (9) with more shaping items has led to better learning speed, thanks to the reward gradient created by the shaping items. When the gradient's effect expands to the point where the reward field is distorted, the success rate falls significantly through a threshold 0.9 (as shown in Fig. 9), the learning speed drops (Fig. 8 with ω dev = 2.0). One can set an upper limit for shaping the reward function that is determined by threshold 0.9 (or another desired number) of the success rate.
Copy expert for knowledge transfer
When the given domain task is too complex and the sensory information limited, RL may become ineffective and/or inefficient for knowledge capture. In this case study, we investigate how directly copying the knowledge (i.e., neural network) from the expert agent who learned from simpler tasks can help the student agent learn in the closely related but more complex tasks. It is assumed that the expert agent has been fully trained using the source Task A with complexity (A) = 2.345 (Fig. 7 and Table 3), and L-Path2 (Table 2) is employed for student training.
In this study, the first target task is Task B: complexity (B) = 3.126, similarity (A,B) = 0.449. This transfer case can be considered as a “high similarity” case. The second target task is Task D: complexity (D) = 4.359, similarity (A,D) = 0.320. Comparing with Task B, this case can be considered as a “low similarity” case. For the purpose of comparison, two “bootstrap” scenarios are introduced, in which the student agent learns from scratch in the target tasks B and D without receiving any information from the expert agent.
Figure 10 illustrates the results of this case study, with Figure 10a showing the effect of “copy expert” for a high similarity case and Figure 10b for a low similarity one. The plots are all the average values of 10 simulation runs. The spreads in Figure 10b show the large deviation of the 10 simulations runs due to the higher task complexity and lower task similarity. Deviations for simpler tasks A and B are relatively small and are not plotted.
Comparison of “copy expert” transfer learning and “bootstrap” learning from scratch.
As shown in Figure 10, “copy expert” helped the student learn much faster compared with “bootstrap”. When the target task similarity is high (Fig. 10a), the “bootstrap” learner can eventually reach the desired reward level, but “copy expert” transfer improved the learning efficiency significantly. When the similarity is low (Fig. 10b), however, the “bootstrap” learner can hardly reach a desired level of reward or it may take endless time to do so. In this case, “copy expert” played a significant role to increase the learning effectiveness and efficiency for the student agent. While the learning variance is relatively low for low complexity and high similarity target tasks (Liu and Jin, Reference Liu and Jin2018), significant learning variance can be seen in high complexity and low similarity tasks, meaning many student agents could not achieve a higher reward score, especially for the bootstrap agents, as shown in Figure 10b.
The results in Figure 10 indicate that “copy expert” can help increase the learning efficiency and the improvement can be significant for low similarity tasks. However, when the target task becomes more complex and the similarity is low, “copy expert” alone may not be effective enough in guiding the learning process of student agents, as shown in Figure 10b. In the next two case studies, we explore whether and how adding a transfer period to “copy expert” may help learning in more complex and unsimilar target tasks.
Adjust transfer period
In this study, the belief-based transfer RL approach described in the section “Knowledge transfer: a belief-based approach” is applied to train the student agents through L-Path3 (Table 2). Both Task B and Task D are used as target tasks, so that both high similarity and low similarity tasks are covered. The goal here is to investigate how changing transfer duration length may influence the student agent's learning behavior while keeping the transfer belief at β 0 = 0.9. Since the student agent takes learning path L-Path3, it has already copied the neural network from the expert agent that was trained in source task Task A.
Figure 11 illustrates the learning performance of varying transfer period from 150 K to 300 K, 700 K, and 1 M frames with yellow, blue, green, and pink colors, respectively. The target task is Task B, a high similarity task. As described in the section “Knowledge transfer: a belief-based approach”, shorter transfer period T tran means the shorter period of expert help (Fig. 6). From a learning speed point of view, the results in Figure 11 show that longer transfer periods lead to better learning performance, with the effect diminishing as it becomes excessively long (after 700 K frames). When the transfer period is getting closer to 1 million frames, which is also the annealing time of ɛ decreasing to 0.1, the performance decreases. Comparing with the “copy expert” case (the red line in Fig. 11), the positive impact of expert help is considerably large, especially until the 20 K episodes range.
For target Task B (high similarity): average learning performance.
Since the transfer period of 700 K turned out as the best arrangement for transfer learning in the similar target task situations, we applied the same parameter values β 0 = 0.9 and T trans = 700 K to target Task D, a low similarity task. Figure 12a shows the results of the simulation runs, where the baseline case is in blue average and green spread and is based on learning path L-Path2 “copy expert and then learn by self”, while the transfer learning is in red average and brown spread. Unlike the high similarity cases, the effect of adding “transfer period” did not have much effect on the learning behavior of the student agent.
Performance learning for target Task D, low similarity, with β 0 = 0.9.
For T trans = 700 K, during the early stage, the performance is better than the baseline. However, after the transfer period, the learning variance starts to grow. Although the maximum performance (the upper edge of the brown spread) is still better than baseline, many students perform worse than the baseline (lower edge of the brown spread) due to the high level of variance of learning. The average performance is slightly higher than baseline but can hardly be considered as improvement.
For T trans = 300 K, again, during the early stage, the performance is better than the baseline. However, because the transfer period is rather short, the learning speed did not pick up after the transfer period. The learning variance also starts to grow. Although the maximum performance is still better than baseline (shown as the upper edge of the orange spread), many student agents perform worse than the baseline, as indicated by the lower part of the orange spread. The average performance is slightly lower than baseline but not by a large amount. To a certain extent, the short transfer period with high belief level interfered the normal learning and led to the inferior performance due to the dissimilarity of the target task.
Adjust transfer belief level
Because the student transfer learning in more complex and low similarity target tasks is not sensitive to the change of transfer period in the last case study (given β 0 = 0.9), we turn to varying the transfer belief β 0 in this case study. The target task here is Task D. The transfer period T trans is fixed at 700 K, and the baseline case is the same as the one described in the last case. The initial transfer belief is varied from 0.9 to 0.5, 0.3, and 0.1.
Figure 13 illustrates the results of simulation runs of varying initial transfer beliefs, with β 0 being set to 0.9, 0.5, 0.3, and 0.1, shown in Figure 13a–d, respectively. β 0 measures the initial probability of the student agent picking transfer action suggested by the expert network (this probability is linearly decreasing to 0 at the end of transfer period). As shown in Figure 13b–d, the jump-start effect of lower initial transfer belief settings is less obvious compared to β 0 = 0.9. Additionally, when β 0 = 0.5, as shown in Figure 13b, many students perform much better than the baseline after the transfer period, and the learning variance is very low. This pattern is unique and cannot be found in other transfer beliefs (i.e., β 0 = 0.9, 0.3, and 0.1). Another interesting pattern is that higher transfer belief (β 0 = 0.9) helps the student perform much better in the early learning stage for frequently trying transfer actions; however, this does not guarantee a better performance in the later stage. As can be seen in Figure 13a, the average convergence time of β 0 = 0.9 is almost the same as the baseline. On the other hand, though the early stage performance is almost overlapping with the baseline in the cases with smaller transfer beliefs, as time proceeds, the learning starts to differentiate from the baseline. It appears that the value of the initial belief level β 0 has a profound impact on the later part, rather than start-up, of students’ learning process.
Learning performance in response to changing initial beliefs for low similarity tasks.
Conclusions
Both robotic research and transportation industries have dealt with collision avoidance problems for years. In addition to system control methods, the intelligent collision avoidance support is often needed for unforeseeable situations where human-like intelligence is demanded. Recent progress in machine learning, especially RL, has made it possible to train agents to acquire collision avoidance knowledge in the way closely as humans do. The knowledge capture based on RL is still domain and task-dependent. A better understanding is needed on how to shape agents’ learning and task performance, and how to make use of the machine knowledge that has already been acquired through previous machine learning processes.
In this research, both the design of reward functions and the use of previously learned knowledge are investigated in the context of knowledge capture. The exploration of the reward functions revealed the trade-offs between types of shaping functions, and our proposed belief-based transfer RL approach provided useful design parameters to transfer experts’ knowledge in different target task situations. The following are some conclusions.
• Reward function is a useful instrument for designing agent behaviors. It is important to understand the effect of ending/terminal rewards and interim/shaping rewards. Designing interim rewards is an effective way to shape the agent's behavior, provided the sparsity and shape of the reward function are recognized and the trade-off between the performance measures clarified, since unexpected effects may happen.
• “Copy expert” can be a very effective strategy for multi-step learning in approaching complex RL tasks. For high similar target tasks, copy expert can significantly improve the initial learning stage and provide needed jump-start of learning. For more complex and low similarity cases, copy expert can significantly reduce the learning deviation and orient the learning process into a right direction reaching to a much higher level of reward.
• “Consult expert” by adding a transfer period to copy expert can be very helpful for the student agent in transfer learning. For high similarity target tasks, longer transfer period tends to be more effective in ramping learning early, and there appears to be an optimal length of the transfer period, beyond which the effect diminishes and turns negative. For more complex target tasks with low similarity, consult expert does not have a significant effect in providing a further improvement on top of copy expert.
• For more complex and low similarity target tasks, adjusting initial transfer belief is a useful way to gain further improvement. While higher transfer belief level helps high similarity target tasks, for low similarity target tasks, lower belief level works better since the expert's old knowledge is less relevant in the new target task. Instead of having a “jump-start” effect, reducing the level of transfer belief helps accelerate learning at a later time after the transfer period is passed. There appears to be a best value of transfer belief where the learning is most effective, and the variation the least.
It should be noted that the complexity and similarity measures, the simulation results and the insights discussed above are limited to the types of the tasks explored in this research. Further research is needed in order to draw more general conclusions. Our ongoing research attempts to expand the generality of this work by exploring more design parameters and more case studies. Another direction along this research line is to explore multi-agent transfer reinforcement learning in the domains of autonomous collision avoidance for ships, cars, and industrial robots.
Acknowledgments
The authors are grateful to the Monohakobi Research Institute (MTI) team, led by Dr. Hideyuki Ando, for their support and collaboration on this research. Thanks are also extended to Mr. Edwin Williams for his valuable comments and suggestions during the course of this research.
Funding
This paper was based on the work supported by MTI and Nippon Yusen Kaisha (NYK). The authors are grateful to MTI and NYK for their kind support.
Xiongqing Liu received his PhD degree in Mechanical Engineering from University of Southern California in the summer of 2019. His research was focused on developing transfer reinforcement learning algorithms for robotic and vehicle collision avoidance. By combining reinforcement learning with transfer learning, Dr. Liu introduced a belief-based approach to enable the transfer of knowledge obtained from reinforcement learning in one task domain to another similar task domain. The approach allows engineers to adjust the belief level and transfer period for the best transfer effect depending on the similarity levels of the source and target tasks. Dr. Liu received his BS degree from Shanghai Jiaotong University and is currently working as a system engineer in the machine learning related areas.
Yan Jin is Professor of Aerospace and Mechanical Engineering at University of Southern California. He received his Ph.D. degree in Naval Engineering from the University of Tokyo and did his post-doctoral research at Stanford University. Dr. Jin's research covers design theory & methods, multiagent and self-organizing systems. His current research interests include design cognition, machine learning and its application in engineering design, knowledge capturing, self-organizing and adaptive systems. Dr. Jin is a recipient of National Science Foundation CAREER Award and TRW Excellence in Teaching Award. He served as Editor-in-Chief of AIEDAM and Associate Editor of JMD and is currently Associate Editor of Design Science Journal. Dr. Jin is a Fellow of ASME.
