1. Introduction
With the increasing prevalence of embodied intelligence technology, people have posted a higher requirement on the service capabilities of robots. To better serve daily life and assist humans in managing transactional tasks, the robot needs to incorporate AI models with their inherent mobility and manipulability to cultivate the ability to interact with the environment. For instance, as illustrated in Figure 1, when we ask the robot to retrieve a lemon from the refrigerator, the robot will open the refrigerator and attempt to grasp the target object. Nevertheless, the target object “lemon” may be obstructed or absent, i.e., the object is invisible in the current constrained environment. This situation requires the robot to utilize its current perceptual data and historical knowledge to make exploration decisions and manipulate objects within the constrained environment, thus retrieving the target object.
Overview of the embodied exploration and manipulation task. This task necessitates that the robot should fully utilize its decision-making and manipulation abilities to retrieve the object required by the human.
Figure 1. Long description
The image consists of one photo and one diagram. The photo shows a robot arm reaching into an open refrigerator to fetch a lemon. The diagram provides close-up views of the refrigerator's contents from first-person and third-person perspectives. The first-person perspective shows various items on the top shelf, including bottles, cans, and a box. The third-person perspective highlights the target object, a green can, located on the middle shelf. The combined visuals illustrate the robot's task of retrieving a specific item from the refrigerator.
To achieve better exploration, more and more researchers have attempted to employ large language models (LLMs) in the decision-making process [Reference Hu, Lucchetti, Schlesinger, Saxena, Freeman, Modak, Guha and Biswas12, Reference Ren, He, Girshick and Sun24]. Based on the perception results obtained from well-trained perception models, such as Faster RCNN [Reference Singh, Blukis, Mousavian, Goyal, Xu, Tremblay, Fox, Thomason and Garg27], LLMs directly generate decision-making sequences to interpret human intentions. Although LLMs are capable of generating the decision-making sequences for object exploration in specific scenarios, these approaches rely solely on the inherent prior knowledge of LLMs. Due to the lack of in-depth analysis of relevant knowledge, such as historical experience and current perception content, the decision-making process of the model may exhibit certain biases [Reference Sun, Du, Ding, Ma, Zhao, Qiu, Liu and Qin28, Reference Tan, Zhang, Liu, Zheng, Wang and An29]. Meanwhile, these decisions are typically accompanied by the robot’s manipulation of various objects, which facilitate the interaction between the robot and the environment. Consequently, the robot can continuously update its historical experience and perception content.
With the advancement of deep learning, current grasp detection techniques have achieved great progress [Reference Kumra, Joshi and Sahin16, Reference Morrison, Corke and Leitner19, Reference Wang, Zhou and Kan30]. Existing methods primarily focus on designing model architectures to improve the accuracy of the grasping detection process or reduce the time required for grasping detection. However, they struggle to balance between accuracy and speed. As a result, robots are incapable of grasping target objects in a robust manner. This limitation hinders their practical application in daily scenarios.
In light of the critical considerations outlined above, in this paper, we develop a novel embodied exploration and manipulation framework augmented with knowledge. This framework aims to empower the robot with the ability to explore and grasp the objects required by humans within a constrained environment. It mainly consists of three modules: embodied exploration module, robotic grasping detection module, and grasping execution module. More specifically, the first module enables the robot to actively explore and interact with the complex environments. To achieve this, we introduce a multi-source knowledge guided decision-making module based on the LLMs. On the one hand, the integration of multi-source knowledge can enhance the perception ability of the robot. On the other hand, the inherent decision-making biases of LLMs can be alleviated. Meanwhile, the robot needs to manipulate the objects in the current scene to realize the continuous interaction. Therefore, we integrate a novel robotic grasping detection network with these exploration sequences. Specifically, to improve the accuracy and efficiency of the grasping detection process simultaneously, a two-branch feature fusion module with a residual block is designed. This network is built upon the generative convolutional neural network. Consequently, conditioned on the predicted grasping poses, the robot can transform them into its own coordinate system to achieve the final object manipulation.
The main contributions of this paper are summarized as follows:
-
• We develop a novel knowledge-augmented embodied exploration and manipulation framework for robots.
-
• We propose to incorporate the scene-relevant knowledge and object correlation coefficient with the LLMs to generate robust and efficient exploration sequences.
-
• We propose an efficient and accurate grasping detection network designed to help robots achieve effective operations.
-
• Extensive experimental results on both simulation and real-world demonstrate the effectiveness and validity of the proposed framework.
2. Related work
In this section, we categorize and discuss studies closely related to our work into the following groups, i.e., embodied exploration and grasping detection.
2.1. Embodied exploration
Embodied exploration demands that a robot actively engage in environmental exploration and learn to interact with the environment by fully utilizing its own capabilities. This approach has been widely studied within the research domain of robotic navigation [Reference Bigazzi, Cornia, Cascianelli, Baraldi and Cucchiara1, Reference Jing and Kong15, Reference Ramakrishnan, Jayaraman and Grauman22, Reference Ren, Clark, Dixit, Itkina, Majumdar and Sadigh23, Reference Sima, Tan, Liu, Sun, Xu and Fu26]. For instance, Ren et al. [Reference Ren, Clark, Dixit, Itkina, Majumdar and Sadigh23] provide a comprehensive analysis of the critical role of robotic, embodied visual exploration in navigation and other aspects. They propose a standardized framework for classifying and benchmarking current embodied visual exploration algorithms and conduct in-depth evaluations by combining two highly realistic simulated 3D environments.
However, relatively few studies have considered integrating this technology into the robotic operation process [Reference Breyer, Ott, Siegwart and Chung2, Reference De Farias, Marturi, Stolkin and Bekiroglu7, Reference Deng, Sima and Liu9, Reference Morrison, Corke and Leitner20, Reference Saito, Ogata, Funabashi, Mori and Sugano25, Reference Zhang, Wang, Han, Li, Zhao, Wang, Duan, Fang, Li and He34]. Morrison et al. [Reference Morrison, Corke and Leitner20] propose to actively perceive the environment to select informative viewpoints for the subsequent grasping process. Deng et al. [Reference Deng, Sima and Liu9] introduce a dynamic scene graph to represent the spatial relationships between objects during the embodied exploration process. Meanwhile, they also propose a GRU-based structure to address the sequence-to-sequence task of manipulation planning, and the scene graph updates conditioned on the outcomes. Recently, LLMs have demonstrated significant promise in decision-making for robots [Reference Hu, Lin, Zhang, Yi and Gao11, Reference Jin, Li, Yong, Shi, Hao, Sun, Zhang and Fang14], showcasing potential advantages in embodied exploration. Despite the fact that these LLMs possess vast knowledge, they still tend to make biased predictions.
2.2. Grasping detection
Grasping detection aims to predict the grasping postures of a given object for subsequent robotic manipulation tasks. With the advancement of deep learning, significant progress has been made in grasping detection [Reference Kumra, Joshi and Sahin16, Reference Wang, Zhou and Kan30, Reference Yu, Zhai, Xia, Wu and Liao32, Reference Zhai, Yu and Xia33]. Kumra et al. [Reference Kumra, Joshi and Sahin16] conceptualize the grasp posture detection process as a generative model and propose a novel generative residual convolutional neural network conditioned on deep convolutional neural modules. Similarly, Morrison et al. [Reference Morrison, Corke and Leitner19] introduce a lightweight grasping detection network capable of one-to-one mapping from depth images. Although their approach achieves rapid prediction and can be used in a closed-loop system, it struggles to deliver consistently strong performance. Building upon these approaches, Yu et al. [Reference Yu, Zhai, Xia, Wu and Liao32] further integrate the UNet and residual attention modules into the grasp detection process.
In contrast, leveraging the advancements in attention mechanisms within Transformer models, Wang et al. [Reference Wang, Zhou and Kan30] propose a transformer-based architecture for robotic grasp detection, featuring two important innovations: window attention and a hierarchical encoder-decoder structure. However, these methods face challenges in balancing the accuracy and efficiency of grasp detection, making them difficult to implement in real-world applications. Recently, there have been several studies on the manipulation of cluttered scenes [Reference Chen, Zhang and Tu4, Reference Kundu, Dutta and Kumar17, Reference Xi, Li and Liu31]. However, these studies have focused on improving efficiency in planar manipulation while neglecting manipulation in constrained spaces.
3. Methodology
In this paper, we aim to endow service robots with the ability to actively explore and manipulate within constrained environments, integrating knowledge and LLMs to efficiently tackle a range of tasks required by humans. This task firstly requires the robot to comprehend the human instruction
$\mathcal{L}_{\pi }$
and analyze to identify the target object
$t_{o}$
. Subsequently, based on the environmental perception content
$\mathcal{S}$
, the robot incorporates the relevant knowledge
$\mathcal{K}{\pi }$
to generate exploration plans
$\mathcal{O}{\pi }$
along with corresponding grasp actions
$\mathcal{G}_{\pi }$
. The entire process can be formally represented as
$\{\mathcal{O}_{\pi }, \mathcal{G}_{\pi } \}=f_{\theta }(t_{o}|\mathcal{L}_{\pi },\mathcal{K}_{\pi },\mathcal{S})$
, where
$f_{\theta }$
denotes the planner
$f$
with parameter
$\theta$
.
An overview of our proposed framework is illustrated in Figure 2, which consists of three main components, i.e., (1) embodied exploration module, which determines the exploration sequences for the objects in the current environment; (2) robotic grasping detection module, which generates the grasping poses for the objects; and (3) grasping execution module, which enables the robot to grasp the objects. The details of each module in our framework will be introduced in the subsequent sections.
An overview of the proposed embodied exploration and manipulation framework, mainly consisting of three modules. One module exploits decision-making for the exploration with multi-source knowledge and LLMs. Another module concentrates on improving the accuracy and efficiency of the grasping detection network. The remaining module is responsible for executing the grasping actions.
3.1. Embodied exploration
In complex operational environments, the sequence in which a robot explores various objects significantly influences the efficiency of task execution. Current decision-making methodologies based on LLMs are constrained by prior training data, which may introduce biases in exploration sequences without comprehensive consideration of the current environmental context. As demonstrated in Figure 2, when instructing a robot to retrieve a strawberry that is not visible in the current robot view, LLM-based approaches may assume its position to be behind a fruit (e.g., a red apple) due to its inherent biases [Reference Sun, Du, Ding, Ma, Zhao, Qiu, Liu and Qin28, Reference Tan, Zhang, Liu, Zheng, Wang and An29], disregarding the discrete spatial distribution characteristic of objects within the refrigerator. Specifically, the strawberry may be obscured by larger objects, not exclusively apples. Furthermore, the dense clustering of apples creates spatial constraints incompatible with strawberry storage requirements. These observations substantiate the weak spatial correlation between strawberries and apples within such configurations. To address these limitations, we propose a multi-source knowledge fusion embodied exploration module designed to mitigate decision-making biases inherent in LLM-based systems. The multi-source knowledge includes domain expert knowledge, historical operation data, and current scene perception information. The architectural implementation is detailed in Figure 3.
The architecture of the embodied exploration module. The target object is the strawberry, which is not visible in the current scene.
Given the current perception RGB image
$\mathcal{I}\in \mathcal{S}$
, we first utilize one pre-trained Faster RCNN to extract object-level feature
$F_{{o}}$
as local context for the detected object and global-level feature
$F_{\mathcal{I}}$
as global context. Notably, the required object (i.e., strawberry) is not visible. Consequently, we retrieve the features from a pre-built knowledge pool
$\{t_{o}\;:\;\{F^k_{t_{o}}\},\cdots \}$
, comprising object keys
$t_o$
and corresponding feature values
$F^k_{t_{o}}$
. The knowledge pool is utilized to provide the relevant features extracted from diverse scenes via a pre-trained Faster RCNN for the specific object. Subsequently, specific object-level features can be retrieved from the knowledge pool via a feature fusion component. The process can be formulated as:
\begin{equation} F_{{t_{o}}}^k=\text{MLP}\big(F^k_{t_{o1}},F^{k}_{t_{o1}},\cdots ,F^{k}_{t_{oi}}\big),\quad i\in \mathbb{N} \end{equation}
Conditioned on the relevant knowledge features and local and global context, we develop a correlation analysis component to address the issue of bias in the inherent exploration sequence. The correlation module is composed of three fully connected layers. Specifically, five participants were systematically recruited to arrange objects randomly within refrigerators across simulated and real-world environments. These participants were instructed to identify specific target objects for retrieval and their corresponding associated items. Subsequently, perceptual imagery of target objects was integrated with their co-occurring items as training datasets, wherein associated objects served as positive labels
$\mathcal{Y} \in \{0,1\}$
for a binary classification model to evaluate relational proximity among objects. Eq. (1) indicates the target object is behind the object that needs to be manipulated. The training process is formulated as:
\begin{equation} \underset {\theta }{min}=\text{BCE}(\text{MLP}(\text{MLP}(F_{\mathcal{I}}),\text{MLP}(F_{o}),F_{{t_{o}}}^k),\mathcal{Y}) \end{equation}
Upon adequate training, the correlation analysis module can quantitatively assess the degree of association between individual detected objects within the image and the target object. Notably, Eq. (2) is trained on all the objects. Empirically, regions exhibiting higher object density demonstrate increased probability of target occlusion. To leverage this expert knowledge effectively, we employ Intersection over Union (IoU) bounding boxes to assess the local density surrounding each object. Specifically, when the condition IoU(
$t_{o1}\cap t_{o2})\gt 0$
is satisfied, the density value associated with
$t_{o1}$
is incremented by one.
By integrating the candidate object list, correlation coefficients, and expert knowledge, LLMs are capable of generating unbiased exploration sequences. Additionally, to fully utilize prior exploration knowledge, we integrate historical knowledge into the decision-making process.
3.2. Robotic grasping detection
To improve the effectiveness and efficiency of the grasping detection process, we propose a novel lightweight grasping detection network, as illustrated in Figure 4, consisting of three components, i.e., a visual encoder, two-branch feature fusion, and a feature decoder.
The architecture of the robotic grasping detection module.
Figure 4. Long description
The diagram represents the architecture of a robotic grasping detection module. It includes a visual encoder, two-branch feature fusion, and a feature decoder. The process starts with an image of an object, which is processed by the visual encoder. The visual encoder consists of a series of Conv 2D and Batch Normalization layers. The output from the visual encoder is fed into the two-branch feature fusion module, which includes several Conv 2D layers and element-wise operations. The fused features are then passed to the feature decoder, which also consists of Conv 2D and Batch Normalization layers, and includes a residual block. The feature decoder outputs multiple heatmaps and a final image of the object with a highlighted grasp point.
Specifically, following previous methods [Reference Kumra, Joshi and Sahin16, Reference Liu, Xie, Huang, Wang and Zhou18], we employ a similar visual feature extractor composed of 2D-CNN with various sizes of kernels and batch normalization to extract the latent features
$f_i$
from the given perception images, including RGB and depth information. To comprehensively capture inherent contextual information from these features, we propose a two-branch feature component. One branch incorporates two sequential 2D-CNN layers to derive fine-grained features
$f_{e}$
from
$f_i$
, i.e.,
$f_e$
= Conv2D(Conv2D(
$f_i$
))+
$f_i$
. The other branch is composed of one 2D-CNN to extract coarse-grained features
$f_c$
from
$f_i$
, i.e.,
$f_c$
= Conv2D(
$f_i$
)+
$f_i$
. Subsequently, these two branches are interconnected through a residual structure, enabling the network to effectively leverage both the newly learned features and the original features. Lastly, the two-branch features are combined to obtain the fused features
$f_o$
. The process is formulated as
$f_{o}=\;f_e+f_c$
.
Following the representation update, prior to decoding the fused features, we employ residual blocks to facilitate deep contextual integration by incorporating features from the visual encoder. This approach enables the utilization of more relevant contextual information for the subsequent decoding process. Subsequently, the fused features are fed into the feature decoder, which consists of 2D-CNN and residual blocks, to obtain the pixel-level predictions, including grasping quality
$Q$
, grasping angle Cos2
$\theta$
and Sin2
$\theta$
, and grasping width
$W$
. Additionally, we implement a sigmoid activation function to normalize
$Q$
and
$W$
values within the bounded range [0,1] to optimize prediction accuracy and computational efficiency.
In order to enable the network to accurately learn grasping detection, we introduce supervised loss constraints to train the proposed network. Specifically, we utilize the binary cross-entropy loss function in the following equation to supervise the training process.
\begin{equation} \mathcal{L}_{q/w}=-\frac {1}{M}\sum _{m=0}^{M}\big[y^m_{q/w}\cdot \text{log}\big(p^m_{q/w}+\big(1-y^m_{q/w}\big)\big)\cdot \big(1-\text{log}\big(p^m_{q/w}\big)\big)\big], \end{equation}
where
$M$
denotes the total number of spatial elements (pixels) of the output feature maps.
$y^m_{q/w}$
denotes the ground-truth label at position
$m$
.
$p^m_{q/w}$
denotes the predicted probability at
$m$
-th position. The subscripts
$q$
and
$w$
represent grasping quality and width, respectively. As for the angle, we utilize smooth L1 loss function to supervise the prediction process.
\begin{equation} \mathcal{L}_a=\frac {1}{M}\cdot \sum _{m=0}^M \begin{cases} 0.5(p^m-y^m)^2, & \text{if $|p^m-y^m|\lt 1$} \\ |p^m-y^m|-0.5, & \text{otherwise} \end{cases} \end{equation}
The whole training loss is formulated as follows:
\begin{equation} \mathcal{L}=\mathcal{L}_{q}+\mathcal{L}_{w}+\mathcal{L}_a \end{equation}
3.3. Grasping execution
Based on the grasping quality, grasping angle, and grasping width, the grasping poses for the object can be systematically formulated. Subsequently, these grasping poses will be provided to the robot to execute the grasping actions. The optimal grasp pose
$\mathcal{G}_{\pi }^*=(x^*,y^*,\theta ^*,w^*)$
can be calculated by the following:
\begin{equation} \begin{array}{ll} (x^*, y^*) = \mathop {\arg \max }\limits _{(x,y)}(Q), & \\[5pt] \theta ^* = \dfrac {1}{2}\arctan \dfrac {sin(2\Phi )}{cos(2\Phi )}|_{(x^*, y^*)},& \\[5pt] w^*= W|_{(x^*, y^*)},&\\ \end{array} \end{equation}
Meanwhile, the transform matrices obtained by hand–eye calibration and camera calibration are used to calculate the operation poses
$g_r$
in the world frame: Then, the transformation matrices derived from hand–eye calibration and camera calibration are utilized to compute the operational poses for the object
$g\in \mathcal{G}_{\pi }$
. The process is computed as follows:
\begin{equation} g = T_{cr}(T_{ic}(\mathcal{G}_{\pi }^*)) \end{equation}
where
$T_{ci}$
and
$T_{rc}$
denote transform matrices of the 2D image space to camera space and the camera space to robot space, respectively.
4. Experiments
In this section, we conduct quantitative and qualitative experiments concerning the simulation environment and real-world environment to validate the effectiveness of our proposed method.
4.1. Experimental settings
The number of objects we utilized in the simulation is set to 15. They are adopted from both YCB [Reference Calli, Singh, Walsman, Srinivasa, Abbeel and Dollar3] and Deng [Reference Deng, Guo, Guo, Zhang, Liu and Sun8]. The target object is varied in each trail. The number of objects in the real-world environment is set to 13. The pre-defined scenes include three types according to the number of objects: mild clutters (
$\le 4$
), moderate clutters(
$5\sim 7$
), and severe clutters (
$\ge 8$
). We select ChatGLM [Reference Zeng and Xu10] as the basic decision-making LLM. Notably, in this experiment, we assume that there is no stacking of objects, and the objects are arranged randomly by the participants. The simulation and the real-world experimental environment are shown in Figure 5, where the simulation environment is built upon CoppeliaSim.Footnote
1
The grasping detection network is trained on Ubuntu 20.04 with Intel(R) Core(TM) i9-10850K CPU and a single RTX 3090.
(a) Simulation experiment setup. (b) Real-world experiment setup.
For the performance of the embodied exploration module, we adopt the Exploration Completion Rate (ECR%) and Completion Steps (CS). ECR represents the success rate of completing the exploration, while CS denotes the optimal number of steps taken to achieve successful completion of the exploration. If the number of explorations exceeds 3, it is considered a failure. As for the robotic grasping detection module, we employ the widely utilized metrics [Reference Cheng, Wang and Meng5, Reference Liu, Xie, Huang, Wang and Zhou18] to evaluate the performance of grasping detection, i.e., the angle threshold between the predicted grasp angle and ground truth should be less than
$30^{\circ }$
and the Jaccard index should be higher than 25%. As for the whole framework, we adopt the Task Completion Rate (TCR%) to evaluate the performance.
4.2. Evaluation on embodied exploration
To verify the performance of the embodied exploration method, we conducted comparative experiments with a variety of exploration methods in both simulation and real-world environments. Before conducting an in-depth analysis of the results, we first provide a brief introduction to the compared exploration methods.
-
• Random: Explore objects in a random order.
-
• Sequential: Explore objects in the field of view in a sequence either from left to right or from right to left.
-
• Rule-based: Explore based on the size of objects. In the case of objects with the same size, the exploration order is determined randomly.
-
• Expert-based: Prioritize exploration starting from areas with high density, followed by random exploration.
-
• LLM-based: Conduct exploration directly based on the decision-making sequences that are generated by LLMs.
The results for the simulation and real-world environment are reported in Tables I and Table II, respectively. As can be observed from the results, we have the following observations: (1) our proposed exploration method demonstrates outstanding performance in both simulated and real-world environments, achieving exploration completion in the minimal number of steps required. (2) As the scenarios gradually become more complex, the success rate of exploration completion declines. Furthermore, the number of steps necessary for completion is higher compared to simpler scenarios. (3) The heuristic-based algorithm exhibits commendable performance in mildly cluttered scenarios. However, as the complexity of the scenarios increases, its performance deteriorates to varying extents. In the simulation environment, when confronted with severely cluttered scenarios, both the random-based and sequential-based methods struggle to achieve successful exploration. (4) In comparison to LLM-based methods, the proposed exploration method demonstrates superior robustness.
Results of the embodied exploration in the simulation environment.
Results of the embodied exploration in the real-world environment.
Performance of grasping detection network on Cornell dataset.
4.3. Evaluation on robotic grasping detection
To assess the effectiveness of the grasping detection network proposed in this paper, we conducted a comparison with recent networks using the Cornell dataset [Reference Jiang, Moseson and Saxena13]. The results reported in Table III demonstrate that the proposed network outperforms others in terms of both accuracy (98.9%) and inference speed (13.2 ms) when utilizing RGB-D modality data. Notably, when compared with algorithms utilizing identical input dimensions (
$224\times 224$
pixels), including GR-ConvNet, SE-ResUNet, and VERGNet, the proposed network maintains significant advantages in both metrics. Furthermore, in scenarios where only RGB data is employed, the proposed network attains an accuracy of 96.6%. In comparison to the algorithm proposed by Cheng et al. [Reference Cheng, Wang and Meng6], the proposed algorithm exhibits a significant advantage in accuracy. Although the inference time is marginally longer, it still satisfies the requirements for practical applications.
Comparison of qualitative results on single objects.
Qualitative results on multiple objects.
Figure 7. Long description
The image consists of two rows with five elements each. The top row contains photos of different objects, while the bottom row displays their corresponding heatmaps. Panel A shows a spoon with its heatmap below. Panel B shows a paper plate with its heatmap below. Panel C shows a set of tools including a knife, pliers, and a screwdriver with their heatmap below. Panel D shows a pair of scissors, a key, and a USB drive with their heatmap below. Panel E shows a pair of headphones with its heatmap below. The heatmaps visually represent the areas of interest or focus for each object.
To further verify the rationale of using the scene image to predict grasping postures with our proposed network, we present a comparative analysis of grasping detection performance with GR-ConvNet, GGCNN, and SE-ResUNet in Figure 6. The qualitative results are alongside the grasping quality. As can be observed from the results, our grasping detection network generates more precise grasping poses based on reliable grasping quality and outperforms existing networks. To further verify the rationale of using the scene image to predict grasping postures with our proposed network, we illustrate the grasping results of known objects along with their grasping quality and compare them with those of GR-ConvNet, GGCNN, and SE-ResUNet in Figure 6. As can be observed from the results, compared with other networks, the grasping detection network proposed by us can generate more accurate grasping poses based on reliable grasping quality.
Although the proposed network is exclusively trained on a single-object grasping detection dataset, it is capable of handling multi-object environments by selecting the objects with high grasping quality. As illustrated in Figure 7, comprehensive qualitative evaluations were conducted using a multi-object grasping detection dataset. The generated grasping poses and corresponding quality heatmaps reveal that the proposed network can generate precise grasping postures based on calculated quality scores in complex multi-object scenarios.
Performance of the whole framework. EX denotes the embodied exploration.
Screenshots of robotic exploration and manipulation experiments within a refrigerator environment.
4.4. Real-world cases
To demonstrate the grasping performance of our proposed knowledge-augmented embodied exploration and manipulation framework in practice, we choose to conduct an overall test of the framework in a moderately cluttered real-world environment. We perform a total of 50 attempts for the whole framework. The results reported in Table IV are evaluated by the Task Completion Rate (TCR%). The experimental results demonstrate that the framework incorporating the embodied exploration algorithm proposed in this study achieved 48 successful grasping manipulations with a 96% success rate. Compared to other methods, our approach demonstrates superior performance. Furthermore, when the grasp detection network introduced in this paper is used to verify the overall success rate across different exploration methods, it becomes evident that utilizing LLMs significantly enhances task completion success rates. Nevertheless, due to the existence of its inherent decision-making bias, its performance remains lower than that of the embodied exploration algorithm proposed in this paper.
To further reveal the effectiveness of our framework, we present the qualitative results for the task execution process in Figure 8. The robot first comprehends the human instruction and utilizes the embodied exploration module to generate the exploration sequences. During the exploration process, the robot also needs to manipulate the object conditioned on the robotic grasping detection and execution module, realizing the environment interaction. These results further demonstrate that our proposed framework enables robots to actively explore and manipulate within refrigerator environments.
5. Conclusion
In this paper, we propose a knowledge-augmented robotic embodied exploration and manipulation framework. The framework aims to endow service robots with the ability to actively explore and manipulate within constrained environments, integrating knowledge and LLMs to efficiently tackle a range of tasks required by humans. To reveal the effectiveness of our framework, we conducted extensive experiments in both simulation and real-world environments across different scenario settings, demonstrating the effectiveness and efficiency of our proposed framework. In this research, we employed a 2D grasping detection network to provide prediction results for robotic manipulation. However, this approach has certain limitations, causing collisions in the exploration postures at some positions. In future research, we plan to incorporate 6D pose estimation into the network process to achieve more precise grasping.
Author contributions
LJ conducted the research and completed the manuscript. KS and LBX conducted the real-world experiments. CQW provided guidance and reviewed the manuscript.
Financial support
This work is supported by the National Natural Science Foundation of the China under Grant 62473370 and China Postdoctoral Science Foundation under Grant Number 2025M781675.
Competing interests
The authors declare that no conflicts of interest exist.
Ethical approval
Not applicable.
Use of AI
Since our research focuses on LLM knowledge, in our paper, we incorporate ChatGLM with our designed modules as the decision tool for the robot.
