1. Introduction
6D pose estimation is a pivotal task in computer vision and aims to predict the object’s 6 degrees of freedom information with respect to the camera, including 3D translation and rotation in space. In addition, it is also widely applied in areas such as lunar surface exploration [Reference Peng, Ma, Xu, Jia and Liu1], robotic manipulation [Reference Chen, Niu, Yang, Qian, Li, Wang, Yan and Huang2, Reference Dong and Zhang3], and autonomous driving [Reference Chib and Singh4]. In robotic manipulation, particularly for grasping tasks [Reference Chen, Li, Lu, Wang, Nie and Yang5, Reference Xi, Li and Liu6], pose estimation is crucial for enabling robots to accurately interact with objects in cluttered and dynamic environments [Reference Zhao, He and Lu7]. However, challenges such as occlusions, textureless surfaces [Reference Li, Huang, Ma and Chen8], and complex object geometries often hinder accurate pose estimation of objects.
Traditional 6D pose estimation methods [Reference Lepetit, Moreno-Noguer and Fua9] tend to rely on sparse keypoint matching, where a set of sparse 3D keypoints is predicted, and geometric models, for example, Perspective-n-Point (PnP) algorithms, are used to compute pose parameters. While these methods perform well in scenes with rich geometric information or textures, they struggle with textureless objects, severe occlusions, and large viewpoint changes. Additionally, the keypoint voting process is computationally expensive, making it unsuitable for real-time applications. Recently, deep learning-based 6D pose estimation methods have achieved remarkable developments, which can be categorized into two main approaches. The first is keypoint-based regression methods [Reference Xiang, Schmidt, Narayanan and Fox10], which predict the positions of object keypoints in 2D images using neural networks and estimate the 3D pose through geometric constraints. However, these methods often fail to accurately predict keypoints when objects lack distinctive features or are partially occluded, leading to significant errors in pose estimation. The second is direct regression methods [Reference Liu, Wen, Peng, Lin, Long, Komura and Wang11], where neural networks directly predict the object’s rotation and translation parameters. While these methods mitigate some challenges related to occlusion, they still struggle to model the nonlinearity of the 3D rotation space effectively, often resulting in suboptimal accuracy in complex scenarios.
To further improve pose estimation, the depth images captured by depth cameras are integrated to provide richer geometric information [Reference Nie, Zhao, Chen, Lu, Li and Yang12], significantly enhancing both algorithmic robustness and accuracy. For instance, methods including SAM-6D [Reference Lin, Liu, Lu and Jia13] demonstrated strong performance by effectively utilizing depth information to establish 2D-3D correspondences, even under challenging conditions such as low-texture surfaces and complex lighting. Despite these advantages, handling complex scenes still remains a challenge for RGB-D-based methods. Issues such as inefficiencies in multimodal fusion and difficulties in capturing fine-grained geometric details often limit their effectiveness. Additionally, these methods struggle to establish robust correspondences in cluttered environments, highlighting the need for more effective solutions.
To address these limitations, we propose a novel Neighborhood-aware Graph Aggregation Network (NGANet) for precise 6D pose estimation. It combines Fully Convolutional Networks (FCNs) and Graph Convolutional Networks (GCNs) to establish dense 2D-3D and 3D-3D correspondences, effectively leveraging both local and global geometric information. By integrating the
$K$
-nearest neighbor (KNN) algorithm to build neighborhood relationships in point clouds and applying GCNs to aggregate local geometric features, NGANet captures surface details and topological structures. Furthermore, we introduce a positional encoding attention mechanism to adaptively fuse multimodal features into a unified and spatially coherent representation. Experimental results show that NGANet attains remarkably high accuracy on LINEMOD and Occlusion-LINEMOD datasets, demonstrating remarkable robustness and generalizability in challenging scenarios such as occlusions and complex environments.
The main contributions are formulated as follows:
-
1) We propose an innovative NGANet that establishes dense correspondences in both 2D-3D and 3D-3D spaces. By fully leveraging local and global geometric information, the network effectively overcomes the limitations of traditional methods in handling textureless objects and complex environments, significantly improving the accuracy and robustness of pose estimation.
-
2) We introduce the KNN algorithm to model neighborhood relationships within point clouds and employ GCNs to aggregate local geometric features, thereby capturing the surface details and topological structures of objects. Furthermore, a positional encoding attention mechanism is utilized to achieve adaptive fusion of multimodal features, generating unified and spatially coherent pose-specific representations.
-
3) We verify the proposed method on both the LINEMOD and Occlusion-LINEMOD datasets, which achieves exceptional results. Experimental findings indicate a substantial enhancement in 6D pose estimation accuracy, even under challenging occlusion conditions, while also confirming its robust performance in real-world applications.
2. Related Work
2.1. RGB-based 6D pose estimation
6D pose estimation methods with RGB images can be broadly categorized into two mainstream aspects: direct regression-based methods and correspondence-based methods. Direct regression methods leveraged deep learning models to directly predict 6D poses. For instance, SSD-6D [Reference Kehl, Manhardt, Tombari, Ilic and Navab14] extended object detection frameworks to make a prediction about objects’ translation and rotation, DeepIM [Reference Li, Wang, Ji, Xiang and Fox15] employed Convolutional Neural Network (CNNs) to iteratively optimize projection errors for pose refinement, and PoseNet [Reference Kendall, Grimes and Cipolla16] directly regressed poses from RGB images. However, these methods were prone to sense noise and the nonlinearity of 3D rotations, leading to suboptimal robustness in complex scenes. On the contrary, methods relying on correspondences predicted the keypoint associations between 2D images and 3D models and utilized the PnP algorithm to compute poses. For example, PVNet [Reference Peng, Zhou, Liu, Lin, Huang and Bao17] predicted keypoints by regressing pixel-wise directional vectors, and HybridPose [Reference Song, Song and Huang18] integrated keypoints with geometric features to significantly improve pose estimation accuracy. ZebraPose [Reference Su, Saleh, Fetzer, Rambach, Navab, Busam, Stricker and Tombari19] combined structured stripe encoding technology with deep learning by projecting stripe patterns onto object surfaces and decoding them, significantly improving the reliability of keypoint detection and the accuracy of pose estimation. CheckerPose [Reference Lian and Ling20] proposed a progressive dense keypoint localization method that refined sparse keypoints into dense distributions through multistage predictions, gradually improving localization accuracy. These approaches exhibited greater robustness when handling occlusion and complex geometric scenarios.
2.2. RGB-D-based 6D pose estimation
RGB-D-based approaches further boosted the accuracy and stability of pose estimation by incorporating depth information. Direct pose prediction methods fused RGB and depth features to directly regress poses. For example, DenseFusion [Reference Wang, Xu, Zhu, Martín-Martín, Lu, Fei-Fei and Savarese21] predicted poses through a pixel-level feature fusion module, while ES6D [Reference Mo, Gan, Yokoya and Chen22] introduced a loss function specifically optimized for symmetric objects to enhance efficiency. However, direct prediction methods relied heavily on feature embeddings and lacked interpretability. The keypoint-based methods estimated poses through projection geometry and keypoint matching. For instance, PVN3D [Reference He, Sun, Huang, Liu, Fan and Sun23] utilized depth information as a crucial means for optimizing geometric constraints, after having predicted 3D keypoints, FFB6D [Reference He, Huang, Fan, Chen and Sun24] enhanced multimodal feature utilization efficiency through a feature fusion module, and Deep Fusion Transformer (DFTr) [Reference Zhou, Chen, Xu, Dou and Qin25] employed cross-modal feature fusion and a weighted vector voting strategy for cutting down computational expenses and improving the robustness of keypoint localization. These methods exhibited strong adaptability in challenging scenarios, such as complex lighting conditions and textureless objects.
2.3. Graph convolution network
GCN has demonstrated strong capabilities in modeling relationships and dependencies in non-Euclidean data, making them well-suited for tasks involving irregular structures like graphs or point clouds. For instance, DGCNN [Reference Phan, Nguyen, Nguyen and Bui26] extracted local geometric features from point clouds to capture fine-grained spatial relationships. SuperGlue [Reference Sarlin, DeTone, Malisiewicz and Rabinovich27] applied GCNs for feature matching by identifying correspondences and filtering out nonmatchable points, showcasing their effectiveness in feature association tasks. PR-GCN [Reference Zhou, Wang, Chen and Huang28] leveraged graph structures to refine 6D pose predictions by enhancing depth representation and constructing local graphs using KNN. RCVPose [Reference Wu, Zand, Etemad and Greenspan29] further integrated graph-based reasoning to effectively fuse RGB and depth features, significantly improving pose estimation performance. SMOC-Net [Reference Tan and Dong30] utilized GCNs to establish robust local and global geometric correspondences, enabling accurate pose predictions even under challenging conditions. Additionally, AM-GCN [Reference Zhong, Lu, Chen, Song and Wang31] introduced an adaptive multichannel architecture to jointly extract specific and shared embeddings from node features and graph topology, further enhancing the representational power of GCNs. These advancements illustrated the versatility and effectiveness of GCNs [Reference Huang, Xu, Bai, Wang and Bai32] across diverse applications, particularly in capturing geometric and spatial information critical for robust pose estimation and feature matching.
3. Methodology
Given RGB-D images, the task of 6D pose estimation is to find a set of rigid transformations that convert the object’s coordinate system
$\mathcal{Z}_b$
into the camera’s coordinate system
$\mathcal{Z}_c$
. This transformation involves two components: a 3D rotation matrix
${\boldsymbol {R}}$
from the special orthogonal group
$SO(3)$
and a translation vector
${\boldsymbol{T}}$
in
$\mathbb{R}^3$
.
3.1. Overview
Our main innovation is the construction of the NGANet. Guided by domain-specific information, NGANet establishes robust correspondences in both 2D-3D and 3D-3D spaces. Unlike Pseudo-Siamese Graph Matching Network (PSGMN) [Reference Wu, Chen, He and Jiang33], our NGANet adopts a multibranch structure that integrates RGB features with both local geometric information from point clouds and global geometric information from mesh data. This design allows the network to simultaneously aggregate fine-grained surface details and overall topological features for precise and robust pose estimation. The architecture of NGANet is shown in Figure 1, including three successive stages: image feature extraction, geometric feature extraction, and pose estimation.
Overall architecture of the proposed NGANet. The image feature extraction stage represents the process of extracting RGB image features, and the geometric feature extraction stage handles point cloud and mesh feature extraction using the SplineCNN structure. The pose estimation stage calculates the 6D pose of the object from 2D–3D correspondences using PnP and RANdom SAmple Consensus (RANSAC).
Specifically, image feature extraction is first responsible for extracting pixel-wise semantic and local RGB features from the input RGB image using FCN. FCN outputs
$X\boldsymbol{}=\{\boldsymbol{H}_{\boldsymbol{s}},\boldsymbol{M}_{\boldsymbol{s}}\}$
, where
${\boldsymbol{M}}_{\textrm{s}}\in \mathbb{R}^{(D+1)\times H\times W}$
denotes the class probabilities for pixel-wise semantic segmentation,
$D$
is the number of object classes, and the additional dimension represents background. The height and width of the 2D image are represented by
$H$
and
$W$
, respectively.
$\boldsymbol{H}_{\textrm{s}}\in \mathbb{R}^{C\times H\times W}$
contains high-dimensional local features for each pixel, where
$C$
denotes the number of feature channels.
Then, the geometric feature extraction aims to establish dense 3D-3D correspondences. It involves constructing a graph
$\mathcal{G}_{c}$
for point cloud data with the application of the KNN algorithm [Reference Steinbach and Tan34] and utilizing the inherent graph structure
$\mathcal{G}_{m}$
for mesh data. Both graphs are processed through GCNs to extract local geometric features
$T_c$
from the point cloud and global topological features
$T_m$
from the mesh. The Neighborhood-aware Cross Attention module (NaCAM) integrates these features by applying position encoding to obtain
$\boldsymbol{P}_c$
and
$\boldsymbol{P}_m$
, followed by a cross-attention mechanism to fuse local and global features, resulting in a unified representation for downstream pose estimation tasks.
In the last pose estimation stage, the fused features are compared with RGB features to compute similarity, which establishes the 2D-3D correspondences. Based on these correspondences, we apply the PnP algorithm to predict the object’s pose, followed by RANSAC for outlier rejection, ensuring robust and accurate 6D pose estimation.
Note that compared with traditional methods using point cloud information, our NGANet additionally introduces the corresponding mesh data to provide topological structure information of the object’s surface. Since point cloud can only describe the discrete point-level features, the triangular patches in mesh are able to effectively define an object’s shape and symmetry, thus compensating for the lack of local geometric correlations in sparse point cloud data. In addition, when combined with GCN, the global topological information from the continuity features of meshes, such as smooth surfaces and edge connections, can be efficiently extracted and serve as prior knowledge to further highlight pose-related features in occluded scenes. Therefore, incorporating mesh information not only strengthens the model’s understanding of the object’s geometry but also improves the accuracy and robustness of pose estimation in complex scenarios through multimodal fusion.
3.2. Graph structure construction
Given an input depth map, it is processed using PointNet++ [Reference Qi, Yi, Su and Guibas35] to generate a point cloud
$P=\{p_i|p_i\in \mathbb{R}^3\}_{i=1}^N$
, which includes both the point cloud features and the 3D coordinate
$(x,y,z)$
of each point
$p_i$
. The coordinates
$P_{\textrm{coord}}$
of each point are extracted from the point cloud features:
\begin{equation} P_{\textrm{coord}}=G({\Phi _\gamma (P)}), \quad \mathcal{F}_{P}={\Phi _\gamma (P)}, \end{equation}
where
$P$
denotes input point cloud,
$\Phi _{\gamma }$
is point cloud feature extraction model with parameters
$\gamma$
, and
$G$
denotes the spatial coordinates extraction function.
To incorporate local neighborhood information, we employ a KNN [Reference Steinbach and Tan34]-based graph construction method. This approach connects each data point to its
$K$
most similar neighbors through edge linkage, forming a sparse yet representative graph structure that preserves local relationships while resisting noise interference. Through extensive experimental validation across multiple datasets and parameter sensitivity analysis, we set
$K$
to 3 by trial and error, leading to the optimal performance. It operates on the extracted point cloud coordinates
$P_{\textrm{coord}}$
and point cloud features
$\mathcal{F}_{P}$
to construct a directed graph
$\mathcal{G}_{c}=(\mathcal{V}_{c},\mathcal{E}_{c})$
. Specifically,
$\mathcal{V}_{c}$
represents points within the point cloud and each point’s connection to its KNN is realized through the edges
$\mathcal{E}_{c}$
, which serve to establish these proximity-based links. The edge indices are defined as:
\begin{equation} \mathcal{E}_{ij} = \{(i, j) \,|\, j \in \arg \min _{j \neq i} \, \mathcal{K}(p_i, p_j), \, \forall j \in P_{\textrm{coord}}, \, |\, j | = k \}, \end{equation}
where
$\mathcal{K}(p_i, p_j)$
is a kernel function used to compute the Euclidean distance similarity between
$p_i$
and
$p_j$
.
$\mathcal{E}_{ij}$
denotes the set of directed edges, with each edge
$(i, j)$
connecting a point
$p_i$
to its
$k$
nearest neighbors
$p_j$
based on the kernel.
Based on the computed edge indices, the edge attributes are derived from the spatial relationships between points as follows:
\begin{equation} \mathcal{A}_{ij} = \phi (p_i, p_j) = \textrm{MLP}\left ([\mathcal{K}(p_i, p_j); \, p_j - p_i]\right ) \! , \end{equation}
where
$\textrm{MLP}(\! \cdot \!)$
is a multilayer perceptron that encodes both the kernel similarity
$\mathcal{K}(p_i, p_j)$
and the displacement vector
$p_j - p_i$
into high-dimensional features.
The constructed graph
$\mathcal{G}_{c}$
, containing node features, edge indices
$\mathcal{E}_{ij}$
, and edge attributes
$\mathcal{A}_{ij}$
, is processed using a GCN designed for irregularly structured data. By leveraging the directed graph structure and incorporating edge attributes, the model achieves robust feature representation for downstream tasks.
3.3. Mesh feature and point cloud feature extraction
In order to process the mesh model and point cloud data using graph convolution, the corresponding graph structure representation needs to be first obtained. A 3D mesh model is inherently a graph structure, as it consists of multiple triangular patches represented in a non-Euclidean manner. Similarly, the 3D point cloud data, represented as a group of unordered points with geometric and feature information, can be naturally processed as a graph structure due to its irregular nature. Therefore, we can obtain the corresponding graph structure representation as
$\mathcal{G} = (\mathcal{V}_m, \mathcal{E}_m, \mathcal{F}_m, \boldsymbol{U}_m)$
, where
$\mathcal{V}_m$
denotes the set of
$N$
vertices and
$\mathcal{E}_m \subseteq {\mathcal{V}_m\times \mathcal{V}_m}$
denotes the set of edges. The node feature matrix
$\mathcal{F}_m$
is composed of the features of each node. The pseudo-coordinate matrix for all nodes is represented as
$\boldsymbol{U}_m$
, where the directed edges formed by these nodes are denoted as
$(i,j) \subseteq \mathcal{E}_m$
, with their pseudo-coordinates represented as
$\boldsymbol{u}(i,j)$
. Additionally,
$\mathcal{N}(i)$
denotes the set of nearby points to node
$i \in \mathcal{V}_m$
.
In terms of point cloud data, the input consists of point cloud features
$\mathcal{F}_{P}\in \mathbb{R}^{|\mathcal{V}_{c}|\times C}$
, where
$C$
represents the combined feature dimensions, including
$x,y,z$
coordinates and learned features. Since each point cloud node has spatial coordinates, we consider using the KNN algorithm to determine the KNN based on their spatial coordinates and construct the edge set
$\mathcal{E}_c$
. The edge attributes
$\boldsymbol{u}_c(i,j)$
are defined as the relative position vectors
$\textrm{A}_{ij}$
between nodes
$i$
and
$j$
, while
$\boldsymbol{U}_c$
represents the complete collection of edge attributes
$\boldsymbol{u}_c(i,j)$
in the point cloud. We utilize multilayer SplineCNN [Reference Fey, Lenssen, Weichert and Müller36] for deep feature extraction from point clouds and mesh models. This network is a GCN that leverages continuous convolutional kernels based on B-spline basis functions. This network aggregates the graph representation of point cloud and mesh, enabling unified feature extraction. The mesh feature extraction process is formalized as
$T_{m}=\Gamma _{\theta _2}(X_n,\mathcal{E}_m,\boldsymbol{U}_m)$
, while the point cloud feature extraction process is expressed as
$T_{c}=\Gamma _{\theta _3}(\mathcal{F}_{P},\mathcal{E}_c,\boldsymbol{U}_c)$
. The use of B-spline-based kernels offers significant advantages, including efficient computation and improved scalability of graph convolution operations. Furthermore, it is worth noting that other well-established GCN models can also be employed in this sub-module to extract mesh features, depending on the specific requirements of the task. In SplineCNN, as for the node
$i$
, the definition of the spatial convolution operation is as follows:
\begin{equation} (\boldsymbol{f}\star \boldsymbol{g})(i)=\frac {1}{\mid \mathcal{N}(i)\mid }\sum _{l=1}^{s}\sum _{j\in \mathcal{N}(i)}f_{l}(j)\cdot g_{l}(\boldsymbol{u}(i,j)), \end{equation}
with
\begin{equation} g_l(\boldsymbol{u})=\sum _{\boldsymbol{r}\in \mathcal{R}}w_{\boldsymbol{q},l}\cdot B_{\boldsymbol{q}}(\boldsymbol{u}), \quad B_{\boldsymbol{q}}(\boldsymbol{u})=\prod _{k=1}^{a}N_{k,q_{k}}^{m}(u_{k}), \end{equation}
where
$f_{l}(j)$
precisely represents the value carried by the
$l$
th dimension of the node feature vector
$\boldsymbol{f}_j$
, and this dimension value plays a crucial role in deeply analyzing the node’s characteristics.
$N_{k,q_{k}}^{m}$
is defined as the
$k$
th open B-spline basis function of degree
$m$
, and it has a close correspondence with the
$q_{k}$
th neighbor. From a set-theoretic perspective, the Cartesian product concept is introduced, with
$\boldsymbol{r}$
being a key element derived from the Cartesian product set
$\mathcal{R} = N_{1,k}^{m} \times \cdots \times N_{d,k}^{m}$
, which consists of multiple B-spline basis functions of the same degree
$m$
. The trainable parameters
$w_{\boldsymbol{q},l}$
determine the weight assigned to each neighboring point
$j$
based on the relative distance
$\boldsymbol{u}(i,j)$
.
$B_{\boldsymbol{q}}(\boldsymbol{u})$
is the product of multiple B-spline basis functions, reflecting the combined effect of the basis functions’ interaction.
$g_l(\boldsymbol{u})$
, as the core computational component, represents the continuous convolution kernel.
In order to make the network implicitly learn the perspective transformation from 3D to 2D, we adopt a multilayer SplineCNN architecture and introduce skip connections to prevent overfitting, enabling the extraction of high-dimensional features from point cloud and 3D mesh models. The final output shapes of
$T_m$
and
$T_c$
are
$C \times \mathcal{V}$
, where
$C$
represents the dimension of node features.
3.4. Local and global geometric features fusion
To integrate point cloud features
$T_{c}=\Gamma _{\theta _3}(\mathcal{F}_{P},\mathcal{E}_c,\boldsymbol{U}_c)$
and mesh features
$T_{m}=\Gamma _{\theta _2}(X_n,\mathcal{E}_m,\boldsymbol{U}_m)$
, we propose the Neighborhood-aware Cross Attention Module using the fusion mechanism based on cross-attention. This module enables the effective integration of local geometric information from point cloud features and global structural information from mesh features. Figure 2 illustrates the basic architecture of the proposed NaCAM.
Fundamental architecture of neighborhood-aware cross attention module. It integrates mesh and point cloud features through a cross-attention mechanism. Mesh features with positional encoding pass through multi-head self-attention, then serve as the query in cross-attention, while point cloud features with positional encoding act as the key and value.
We first encode positional information for both features. Specifically, positional encoding is applied to both point cloud features
$T_c$
and mesh features
$T_m$
using a learnable embedding layer
$\textrm{PE}(\! \cdot \!)$
. The formula is as follows:
\begin{equation} \ \boldsymbol{P}_{c}=\textrm{PE}(T_{c}), \boldsymbol{P}_{m}=\textrm{PE}(T_{m}) \end{equation}
Next, we apply self-attention to the mesh features to enhance their representational capacity in Figure 2. The self-attention mechanism computes the updated mesh features as:
\begin{align}&\qquad \boldsymbol{F}_{m}=\textrm{Attention}(\boldsymbol{Q}_{m}+\boldsymbol{P}_{m},\boldsymbol{K}_{m}+\boldsymbol{P}_{m},\boldsymbol{V}_{m}) \end{align}
\begin{align}&\qquad\,\, \boldsymbol{F}_{c}=\textrm{Attention}(\boldsymbol{Q}_{c}+\boldsymbol{P}_{c},\boldsymbol{K}_{c}+\boldsymbol{P}_{c},\boldsymbol{V}_{c}) \end{align}
\begin{align}& \textrm{Attention}_{\{self,cross\}}(\boldsymbol{Q},\boldsymbol{K},\boldsymbol{V})=\textrm{softmax}\left (\frac {\boldsymbol{Q}\boldsymbol{K}^T}{\sqrt {d_k}}\right )\boldsymbol{V}, \end{align}
where
$\textrm{Attention}_{\{self,cross\}}$
is multi-head self/cross attention structure of standardization from [Reference Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser and Polosukhin37].
$K\in \mathbb{R}^{N\times C}$
and
$V\in \mathbb{R}^{N\times C}$
are direct mappings of
$Q\in \mathbb{R}^{N\times C}$
to input attention structure, and
$C$
is dimension of
$Q$
.
$N$
and
${d}_k$
represent the number of nodes and scaling factor, respectively. The incorporation of their corresponding position embedding is essential when
$Q$
and
$K$
are employed in the attention mechanism.
In order to combine the features from point cloud and mesh, NaCAM employs a cross-attention mechanism. The self-attended mesh features are used as the query
$\boldsymbol{F}_{\textrm{m}}$
, the position-encoded point cloud features serve as the key
$\boldsymbol{F}_{c}$
and the value
$\boldsymbol{F}_{\textrm{c}}$
. It is formulated as follows:
\begin{equation} \boldsymbol{F}_{\textrm{fusion}}=\textrm{Attention}_{\{self,cross\}}(\boldsymbol{F}_{\textrm{m}},\boldsymbol{F}_{\textrm{c}},\boldsymbol{F}_{\textrm{c}}), \end{equation}
where
$\boldsymbol{F}_{\textrm{fusion}}\in \mathbb{R}^{|\mathcal{N}|\times C}$
represents the fused features. After cross-attention fusion, skip connections are applied, followed by the Linear-Relu-Linear layer (LRL) to produce the final fused features.
$\boldsymbol{F}_{\textrm{fusion}}$
establishes dense 3D-3D correspondences by associating mesh feature points with point cloud features that aggregate neighborhood information. The NaCAM module not only captures neighborhood-aware details from the point cloud, but also enhances global consistency through mesh-based queries, achieving a precise and unified representation for pose estimation.
3.5. Pose estimation
The fusion process in NGANet integrates dense 3D-3D correspondences between mesh feature points and point cloud features. These correspondences are established by leveraging the NaCAM, where the fused features
$\boldsymbol{F}_{\textrm{fusion}}\in \mathbb{R}^{|\mathcal{N}|\times C}$
are generated through the cross-attention mechanism. We establish 2D-3D correspondences by calculating the similarity between the RGB features from image and the fused 3D information features. Specifically, given
$\boldsymbol{H}_{s}=\Lambda _{s}(\Psi _{\gamma _{1}}$
$(\boldsymbol{R}))\in \mathbb{R}^{|\mathcal{V}_{m}|\times C}$
and the fused features
$\boldsymbol{F}_{\textrm{fusion}}\in \mathbb{R}^{|\mathcal{N}|\times C}$
, derived from Eq. (10), the feature similarity is computed through a cross-relation operation, which is expressed as:
\begin{equation} {\hat{\boldsymbol{S}}}=Softmax \left(\boldsymbol{H}_s\boldsymbol{F}_{fusion}^\top \right)\in \mathbb{R}^{|\mathcal{V}_m|\times |\mathcal{N}|}, \end{equation}
where the
$Softmax$
function is applied along each row.
$\Lambda _{s}$
specifically gathers the features from
$\boldsymbol{H}_{s}$
corresponding to the parts of
$\boldsymbol{M}_{\textrm{s}}$
that are classified as valid pixels.
$\mid \mathcal{V}_{m}\mid$
represents the number of valid pixels in
$\boldsymbol{M}_{\textrm{s}}$
, reflecting the scale of the image’s features in this regard, while
$\mid \mathcal{N}\mid$
denotes the total number of vertices in
$\boldsymbol{F}_{\textrm{fusion}}$
. The
$i$
th row vector
$\hat {S}_{i,:} \in [0,1]^{|\mathcal{V}_{m}|}$
represents a discrete probability distribution for the potential correspondences of node
$i \in \mid \mathcal{N}\mid$
in
$\boldsymbol{F}_{\textrm{fusion}}$
.
The loss function of our proposed NGANet is defined as:
\begin{equation} \mathcal{L}(\boldsymbol{\theta })=\rho \cdot \textrm{CEloss}(Softmax(\boldsymbol{M}_\boldsymbol{s}),\boldsymbol{M}_\boldsymbol{gt})+\textrm{CEloss}({\hat{\boldsymbol{S}}},\boldsymbol{S}_\boldsymbol{gt}), \end{equation}
where CEloss represents the cross-entropy loss function and
$\boldsymbol{\theta }=\{\theta _1,\theta _2,\theta _3\}$
represents the trainable parameters of the NGANet.
$\rho$
serves as a hyperparameter to control the trade-off between the two components of
$\mathcal{L}(\boldsymbol{\theta })$
. The first term reduces the model’s semantic segmentation error, while the second term ensures accurate matching of dense 2D–3D correspondences. During the training phase, the ground-truth pose is provided, allowing us to project the 3D object model onto the 2D image using the known pose, thereby generating
$\boldsymbol{M}_\boldsymbol{gt}$
. Similarly,
$\boldsymbol{S}_\boldsymbol{gt}$
represents the ground-truth correspondences between the 2D and 3D points.
4. Experiments and results
4.1. Experimental setup
Training details: To facilitate evaluation, both LINEMOD and Occlusion-LINEMOD datasets are partitioned with 85% for training and 15% for testing. To enhance the model’s robustness during training phase, we apply a variety of data augmentation techniques, such as 3D rotations, color jittering, and so on, to original images. The training procedure begins with an initial learning rate of 0.001, which is halved after every 10 epochs. The entire training process lasts for 300 epochs, utilizing two NVIDIA V100 GPUs and the Adam optimizer to optimize the model parameters. In order to balance the contributions from semantic segmentation and correspondence matching, the hyperparameter
$\rho$
is set to 100. This fine-tuning strategy ensures the efficiency and stability of the training, leading to more optimal performance across various task metrics.
Mesh model simplification: To optimize memory consumption, we simplify the 3D mesh models of the objects following approach in [Reference Wu, Chen, He and Jiang33] . With the help of the quadratic edge collapse method implemented in MeshLab [Reference Cignoni, Callieri, Corsini, Dellepiane, Ganovelli and Ranzuglia38], each 3D mesh is processed in a simplified way. Eventually, the number of triangular patches is reduced to less than 8000, and the number of vertices is decreased to less than 4000.
Network structure details: The NGANet is built using PyTorch [Reference Paszke, Gross, Massa, Lerer, Bradbury, Chanan, Killeen, Lin, Gimelshein, Antiga, Desmaison, Kopf, Yang, DeVito, Raison, Tejani, Chilamkurthy, Steiner, Fang, Bai and Chintala39], with the image feature extraction utilizing the ResNet34 architecture from the PyTorch model zoo. Pretrained weights are employed to initialize the image feature extraction process, while the dimensionality of the graph node features is set to 64 to balance computational efficiency and representational capacity.
4.2. Evaluation metrics
To evaluate the effectiveness of different pose estimation approaches, we adopt two benchmark measurement criteria: 1) the point-wise average 2D projection metric [Reference Brachmann, Michel, Krull, Yang, Gumhold and Rother40] and 2) the average 3D distance of model points (ADD) metric [Reference Hinterstoisser, Lepetit, Ilic, Holzer, Bradski, Konolige and Navab41].
2D projection metric: The 2D projection metric is used to evaluate the accuracy of the alignment between a 3D object model and its 2D projection in an image. It primarily involves projecting a 3D object onto a 2D plane and comparing the distance between the projected points of the predicted object pose and the true pose to assess accuracy. It is typically considered that the estimated pose is accurate when the distance between the two poses is less than 5 pixels.
ADD metric: It mainly calculates the average distance between the vertices of the object model transformed by the predicted 6D pose and those transformed by the ground-truth pose. If this value is less than 10% of the model’s diameter, the predicted pose is considered correct. The ADD metric [Reference Hinterstoisser, Lepetit, Ilic, Holzer, Bradski, Konolige and Navab41] involves one set of vertices being transformed by the predicted 6D pose
$[\boldsymbol{R},\boldsymbol{t}]$
and the other set being transformed by the ground-truth pose
$[\boldsymbol{R}^*,\boldsymbol{t}^*]$
.
\begin{equation} \textrm{ADD}=\frac {1}{m}\sum _{x\in \mathcal{O}}\|(Rx+t)-(R^*x+t^*)\|, \end{equation}
where
$x$
represents a vertex from a total of
$m$
vertices on the object mesh
$O$
. The ADD-S [Reference Xiang, Schmidt, Narayanan and Fox10] metric, specifically designed for symmetric objects, calculates the average distance by evaluating the closest point distance between the transformed vertices.
\begin{equation} \textrm{ADD} - \textrm{S}=\frac {1}{m}\sum _{x_1\in \mathcal{O}}min\|(Rx_1+t)-(R^*x_2+t^*)\|k, \end{equation}
where
$x_1$
represents a vertex from the sum of
$m$
vertices on the object mesh
$O$
.
$x_2$
corresponds to
$x_1$
, but is the vertex under the true pose.
$x_2$
is selected such that it minimizes the distance
$\min \|(Rx_1+t)-(R^*x_2+t^*)\|$
among all possible
$x_2$
vertices.
The ADD(S)-0.1d indicator results of different methods under LINEMOD dataset. Symmetrical objects are marked with *, bold values indicate the highest accuracies, and underlined values represent the second-best results.
4.3. Quantitative results
We first evaluate our method with GDR-Net [Reference Wang, Manhardt, Tombari and Ji42], PVNet [Reference Peng, Zhou, Liu, Lin, Huang and Bao17], HybridPose [Reference Song, Song and Huang18], CDPN [Reference Li, Wang and Ji43], PSGMN [Reference Wu, Chen, He and Jiang33], and SMOC-Net [Reference Tan and Dong30], under the LINEMOD dataset. As shown in Table I, our method achieves the highest mean accuracy across all objects under ADD metric, outperforming the best-performing baseline, GDR-Net, by 1.7%. Notably, for the ape object, our method improves the accuracy by 3.8%, and for the phone object, it achieves a significant improvement of 4.9%. The results highlight the limitations of RGB-only regression-based methods, such as GDR-Net, which fail to incorporate geometric constraints from depth information. This limitation is particularly evident for objects such as ape and phone, which often lack rich texture or exhibit complex geometric shapes. By introducing depth information into our method, we effectively enhance the spatial representation, enabling more accurate pose estimation even for textureless or geometrically challenging objects.
The proposed method is further compared with PSGMN [Reference Wu, Chen, He and Jiang33], GDR-Net [Reference Wang, Manhardt, Tombari and Ji42], SO-Pose [Reference Di, Manhardt, Wang, Ji, Navab and Tombari44], SMOC-Net [Reference Tan and Dong30], PVN3D [Reference He, Sun, Huang, Liu, Fan and Sun23], FFB6D [Reference He, Huang, Fan, Chen and Sun24], PR-GCN [Reference Zhou, Wang, Chen and Huang28], RCVPose [Reference Wu, Zand, Etemad and Greenspan29], using Occlusion-LINEMOD dataset. As shown in Table II, our method demonstrates substantial advancements in pose estimation under occluded and cluttered scenes compared to RGB-only methods, which often struggle with limited spatial information. By leveraging depth data and employing a neighborhood-aware cross-attention mechanism for the purpose of integrating local point cloud features and global mesh features, our model effectively captures fine-grained surface details and global structural information, enhancing robustness against occlusion and clutter. However, for objects such as can, glue, and hole puncher, our method is slightly outperformed by SMOC-Net [Reference Tan and Dong30] and FFB6D [Reference He, Huang, Fan, Chen and Sun24], with respective gaps of 6.1%, 7.1%, and 5.4%. This can be mainly attributed to three aspects: 1) These objects contain fine-grained structural details (e.g., can handles and glue caps) that pose significant challenges for detail preservation during feature aggregation. Under occlusion conditions, background interference tends to infiltrate the fusion process, ultimately compromising pose estimation precision. 2) Symmetric objects, such as glue, have geometric properties that increase pose ambiguity, leaving the model struggling to differentiate between symmetric viewpoints when lacking distinct texture or geometric disparities. 3) The complex geometric shapes of objects like the holepuncher demand more effective use of depth information, and in partially occluded scenarios, local information loss may prevent the model from fully modeling global topological relationships. These limitations collectively compromise algorithm robustness on geometrically challenging objects, requiring adaptive neighborhood selection mechanisms and integration of geometric priors to enhance structural modeling capabilities for intricate shapes.
The ADD(S)-0.1d indicator results of different methods under Occlusion-LINEMOD dataset. Symmetrical objects are marked with *, bold values indicate the highest accuracies, and underlined values represent the second-best results.
In comparison to methods based on RGB-D data, our approach also demonstrates clear advantages. Specifically, our method exceeds the state-of-the-art RCVPose [Reference Wu, Zand, Etemad and Greenspan29] by 6.7% in terms of ADD(s)-0.1d. The validity of our feature extraction and combination approach, along with the robustness of our pose estimation pipeline, is demonstrated by this improvement. While RGB-D methods often suffer from suboptimal integration of multimodal information, our approach achieves superior performance through a carefully designed fusion mechanism that ensures coherence across different feature modalities. Hence, our method not only surpasses RGB-based methods in challenging environments but also demonstrates superior precision and resilience compared to state-of-the-art RGB-D-based methods, establishing a new benchmark for pose estimation under occlusion and cluttered conditions.
Illustration of 2D projection metric of our NGANet under LINEMOD and Occlusion-LINEMOD datasets.
In addition, the 2D projection accuracy curves are further given in Figure 3 to show our method’s effectiveness under LINEMOD and Occlusion-LINEMOD datasets. The left subfigure demonstrates that our method attains a continuously high level of accuracy for all objects within the LINEMOD dataset, with rapid convergence after the first few models. It shows that our approach is robust and efficient in situations where there is minimal occlusion and clutter. When it comes to the right subfigure, the accuracy on the Occlusion-LINEMOD dataset remains above 80% for all objects, even under significant occlusion and cluttered backgrounds. Notably, objects such as “Glue” and “Duck” exhibit slightly a lower accuracy due to their narrow or sharp structures, which increase the challenge of establishing precise correspondences. Nevertheless, the overall performance validates our method’s ability to achieve robust and accurate pose estimation in both standard and challenging scenarios.
The computational complexity of different pose estimation methods is finally reported and compared. For RGB-based methods GDR-Net [Reference Wang, Manhardt, Tombari and Ji42] and PSGMN [Reference Wu, Chen, He and Jiang33], their corresponding running times are 22 ms and 40 ms, respectively. When it comes to RGB-D-based methods, PVN3D [Reference He, Sun, Huang, Liu, Fan and Sun23] achieves the inference time of 190 ms due to the processing of depth information, while that for PR-GCN [Reference Zhou, Wang, Chen and Huang28] and FFB6D [Reference He, Huang, Fan, Chen and Sun24] are 68 ms and 75 ms. The running time of our proposed NGANet is about 150 ms, which is less than PVN3D but more than remaining ones, and the increase in computational complexity can be ascribed to two aspects. 1) The time complexity of KNN search and cross-attention mechanism is quadratic relative to the number of nodes. Particularly, when processing large-scale point clouds (about 4000 vertices), the computational demand increases significantly. 2) Graph convolution operations, such as SplineCNN, require substantial resources due to repeated B-spline basis function calculation and neighborhood aggregation. When the topological structure is complicated, dynamic graph update and feature propagation require additional computational power. Considering the concurrent processing of RGB images, point clouds, and mesh data, our method achieves higher evaluation metrics at the expense of a certain drop in running time, which is still acceptable for most pose estimation scenarios.
4.4. Qualitative analysis
The qualitative outcomes in Figure 4 demonstrate the efficacy of our method in estimating 6D object poses on the LINEMOD dataset, particularly for textureless and symmetrical objects. Our approach achieves high pose estimation accuracy across multiple objects, even those without texture, such as the ape and cat, where traditional methods often struggle. Furthermore, for symmetrical objects such as the eggbox and glue, our method excels in disambiguating poses, ensuring precise estimation despite their inherent symmetry. These results highlight the robustness of our model in accurately estimating poses for both textureless and symmetrical objects, leveraging geometric and RGB information to maintain consistent performance in these challenging cases.
Qualitative results of our method under LINEMOD dataset. The red boxes represent the ground-truth poses, while the blue boxes indicate the predicted ones. Each subfigure corresponds to a specific object type from LINEMOD dataset. Specifically, (a) ape, (b) benchvise, (c) cam, (d) cat, (e) driller, (f) duck, (g) holepuncher, (h) lamp, (i) phone, (j) glue, (k) iron, (l) can.
Qualitative results of different methods under Occlusion-LINEMOD datasets. We visualizethe results of PVN3D [Reference He, Sun, Huang, Liu, Fan and Sun23], PSMGN [Reference Wu, Chen, He and Jiang33], and our proposed method in different objects. The red boxes represent the ground-truth poses, while the blue boxes represent the predicted ones.
In Figure 5, the qualitative results under Occlusion-LINEMOD dataset are further given, showing a more remarkable performance of our method than PSGMN [Reference Wu, Chen, He and Jiang33] and PVN3D [Reference He, Sun, Huang, Liu, Fan and Sun23]. Specifically, our method achieves more accurate and robust pose estimations across various objects, even under circumstances such as severe occlusions and cluttered environments. For PSGMN [Reference Wu, Chen, He and Jiang33], notable misalignments can be clearly seen between predicted poses and ground-truth ones, particularly in scenes with occlusions. For example, in the case of “duck” and “eggbox,” the absence of depth information in PSGMN limits its ability to accurately estimate poses in partial occlusion, as missing depth cues prevent the model from properly inferring the correct rotational angles of the objects. This results in substantial deviations in pose predictions, especially when occlusions obscure key features needed for accurate regression. Similarly, while PVN3D [Reference He, Sun, Huang, Liu, Fan and Sun23] performs better than PSGMN, it still struggles with handling complex occlusion scenarios. In particular, for objects such as “cat” and “glue,” which experience truncated occlusions where parts of the objects are hidden, PVN3D faces challenges in accurately regressing their poses. This is due to the difficulty in inferring the complete object shape from partial views, which is further compounded when occluded areas contain key features that are essential for precise pose estimation.
In contrast, our method effectively estimates the poses of objects, maintaining alignment with the ground-truth poses, even in heavily occluded and cluttered scenes. The improvement can be attributed to the neighborhood-aware cross-attention mechanism, which adaptively fuses local point cloud features and global mesh features, enabling our model to better capture geometric relationships and spatial coherence. Furthermore, the incorporation of depth information allows for enhanced geometric reasoning, markedly enhancing the robustness and precision of pose estimation in contrast to those methods that merely depend on RGB features.
Visualization of failure results of our proposed NGANet under Occlusion-LINEMOD dataset. The poses predicted by GDR-net [Reference Wang, Manhardt, Tombari and Ji42] are given for comparison.
In addition to successful results, some failure predictions about pose estimation are also visualized, see Figure 6. Since GDR-Net has open-sourced the code and also outperforms our NGANet on object “can,” we select it for comparison. The results in Figure 6 show that the poses predicted by GDR-Net coincide well with ground truths, but those predicted by our NGANet produce larger deviations. This occurs because neighboring information aggregation mistakenly incorporates points from other objects, particularly in the hollow handle area of the can or under occlusion condition. As for the object “glue,” our method achieves the second-best performance and is only inferior to SMOC-Net. Considering the sharp structures, images from certain angles may cause these structures to overlap with other objects, decreasing the accuracy of our pose estimation approach. In contrast, SMOC-Net is able to achieve more efficient 6D object pose estimation under a self-supervised learning paradigm by leveraging geometric constraints from camera poses and a knowledge distillation framework.
Illustration of real-world pose estimation and its application in robotic grasping. The pose estimation results under nonoccluded (rows 1 to 2) and occluded (row 3) environments are respectively displayed, where red and blue boxes represent ground-truth and predicted poses. In row 4, the real-world robotic grasping procedures using the objects’ predicted poses are given.
4.5. Real-world pose estimation
In order to further verify the validity of our proposed method, the real-world pose estimation experiments in robotic grasping are conducted, where two objects (bottle and banana) are used.
Real-world pose estimation: For a single object nonoccluded scenario, we randomly select one object and place it on the platform, which is imaged by a RealSense D435i camera from different views and scales. As shown in the first and second rows of Figure 7, the objects’ poses can be accurately estimated with our proposed NGANet, indicating its robustness in extracting orientation-invariant pose features. When it comes to multiple objects occluded scenario, although the banana is partially occluded by other objects, its corresponding pose can still be predicted with accuracy, see row 3. It can be ascribed to the GCN for local geometric features aggregation and mesh information for topological structure encoding.
Robotic grasping: With the predicted object pose, we further build a real-world robotic grasping system to verify its validity in object grasping. The system consists of a UR5e robotic arm, a Backyard E140 two-finger gripper, and a platform for placing objects. The D435i camera is vertically installed above the platform. Once the pose information is determined, the 2D grasp rectangles on object are generated using [Reference Chen, Niu, Yang, Qian, Li, Wang, Yan and Huang2], guiding the movement of robotic arm and executing the in-plane grasping. The experimental results in the last row of Figure 7 indicate that the proposed NGANet is capable of performing accurate pose estimation and contributes to successful object grasping in real-world scenarios.
Results of ablation experiments under Occlusion-LINEMOD dataset.
4.6. Ablation study
To verify the significance of incorporating point cloud information and the proposed NaCAM module, we further conduct ablation experiments on the Occlusion-LINEMOD dataset. The experimental results are shown in Table III, where baseline includes RGB and mesh features extraction, feature matching, and pose regression. Specifically, we use the Cosine Similarity-Based Feature Fusion with MLP Dimensionality Reduction module (CoSi-MFP) to integrate point cloud and mesh features. It selects points from the point cloud based on the mesh model using cosine similarity, then concatenates the selected point cloud features with mesh features, followed by dimensionality reduction using MLP. The experiments show that introducing point cloud features improves accuracy by 12.1% compared to the baseline, confirming that the inclusion of point cloud information contributes to pose regression for occluded corner points and edges. However, when objects experience truncated or severely occluded conditions, directly selecting points leads to incorrect or missed selections, significantly affecting the regression accuracy. After incorporating the NaCAM, the accuracy reaches 76.9% (18.7% improvement over CoSi-MFP), indicating that neighborhood-aware information from the point cloud effectively complements objects under truncation and severe occlusion. Moreover, the module dynamically adjusts and learns based on different objects, rather than simply selecting based on vector similarity, which helps avoid misjudgments and missed selections, thereby enhancing the regression accuracy substantially.
5. Conclusion
In this work, we propose a NGANet for precise 6D pose estimation, addressing key limitations of existing methods. By integrating FCNs and GCNs, NGANet effectively establishes dense 2D-3D and 3D-3D correspondences, leveraging both local and global geometric information. The use of the KNN algorithm and GCN-based feature aggregation enables NGANet to capture fine surface details and complex topological structures. Additionally, our positional encoding attention mechanism facilitates adaptive multimodal feature fusion, resulting in spatially coherent representations.
Despite its effectiveness, the proposed method still has certain limitations, and future researches will explore three key directions:
-
1) The proposed method takes static images as inputs, which may suffer from problems of restricted imaging views and hinder the further improvement of pose estimation accuracy. When actively observing the object, how to effectively incorporate temporal information to achieve accurate and stable pose estimation across consecutive frames deserves additional investigation.
-
2) Existing pose estimation methods tend to learn pose-related features in a supervised manner, where large amount of annotated training data is required. To mitigate the cost in collecting and labeling images, how to introduce machine learning strategies, including self-supervised learning, unsupervised learning and semi-supervised learning, and reduce reliance on labeled data can also be studied.
-
3) Currently, the scenarios of our pose estimation experiment, as well as some public datasets, are a bit simple, where single object can be clearly observed. Hence, more complicated real-world variations, such as stacked objects, mutual occlusion, and low-light imaging conditions, may be considered to verify the robustness and adaptability of our method in diverse real-world tasks.
Author contributions
Lu Chen designed and supervised this study. Yuhao Zheng conducted experiments and wrote the article. Peng Wu and Jing Yang performed statistical analyses. Yan Gao and Jingyang Liu collected data and checked the grammar errors in the article. All authors read and approved the final manuscript.
Financial support
This work was supported in part by the National Natural Science Foundation of China under Grant 62373233, 62406181, in part by the Fundamental Research Program of Shanxi Province under Grant 202203021222010, in part by the Research Project Supported by Shanxi Scholarship Council of China under Grant 2023-023, in part by the Science and Technology Major Project of Shanxi Province under Grant 202201020101006, and in part by the Central Guidance for Local Scientific and Technological Development Funds under Grant YDZJSX20231B001.
Competing interests
The authors declare no conflicts of interest exist.
Ethical approval
Not applicable.
