如何将使用EPnP计算的相机姿态变换应用于VTK相机？

Question

对于我的增强现实项目,我使用VTK相机观看3D模型,并使用真实相机观看模型的真实对象.

我使用EPnP来估计真实相机的外在矩阵(这个相机已经预先校准,因此我知道内部参数),通过从真实相机图像和内部参数提供来自VTK及其相应2D点的3D点.用于EPnP算法的真实相机.

之后,我获得了一个旋转和平移矩阵,其元素为 - > R1,R2,R3,.....,R9和t1,t2和t3.

所以我的真实相机的外在矩阵看起来像这样(让我们称之为extrinsicReal)

R1 R2 R3 T1
R4 R5 R6 T2
R7 R8 R9 T3
 0  0  0  1

在此之后,我使用以下代码估计我的VTK相机的外在矩阵:

vtkSmartPointer<vtkMatrix4x4> extrinsicVTK = vtkSmartPointer<vtkMatrix4x4>::New();
extrinsicVTK->DeepCopy(renderer->GetActiveCamera()->GetViewTransformMatrix());

要将VTK摄像机3D模型与真实摄像机融合,VTK摄像机应设置在与真实摄像机位置相同的位置,并且VTK摄像机的焦距应与真实摄像机的焦距相同.另一个重要步骤是将真实相机的相同外在矩阵应用于VTK相机.我该怎么做？

我做的是我采用extrinsicReal的反转并将其与extrinsicVTK相乘得到一个新的4*4矩阵(让我们称之为newMatrix).我将此矩阵应用于VTK相机的转换.

vtkSmartPointer<vtkMatrix4x4> newMatrix = vtkSmartPointer<vtkMatrix4x4>::New();
vtkMatrix4x4::Multiply4x4(extrinsicRealInvert,extrinsicVTK,newMatrix);

vtkSmartPointer<vtkTransform> transform = vtkSmartPointer<vtkTransform>::New();
transform->SetMatrix(NewM); 
transform->Update();

renderer->GetActiveCamera()->ApplyTransform(transform);

我不确定这是否是正确的方法.但是我检查了真实的摄像机位置(我在EPnP之后得到的)和VTK摄像机位置(在应用上面的变换之后)并且它们都完全相同.此外,真实相机的方向和VTK相机的投影方向也是相同的.

问题是即使在上述参数匹配VTK和真实相机之后,3D VTK模型似乎也不能与真实相机视频完美对齐.有人可以指导我一步一步调试问题吗？

Answer 1

将这些参数应用到vtk相机时,事情变得复杂了.我是这样做的(只是重要代码段落的摘录,整个代码在这里粘贴太多了,无论如何都对你没用).其他要考虑的要点:

1. 我在我的vtkRenderWindow中将内窥镜图像渲染为背景纹理.
2. 我使用的是VTK,ITK(vnl),OpenCV函数的混合,但它们应该是可互换的(例如cvRound也可以被vtkMath :: Round()等替换)

首先,我使用vtkRenderer中的有源相机:

d->m_Renderer->GetActiveCamera()

下一步是通过应用变换不断更新活动相机.根据您的渲染窗口是否可调整大小,您必须初始化或不断更新另外两个参数:1.ViewAngle,2.WindowCenter(非常重要,vtk完全没有记录.但最后您必须应用通过校准找到的主要点,或者您将使用偏移量渲染表面.花了3个月才找到这两行解决方案).

视角的计算:

  double focalLengthY = _CameraIntrinsics->GetFocalLengthY();
  if( _WindowSize.height != _ImageSize.height )
  {
    double factor = static_cast<double>(_WindowSize.height)/static_cast<double>(_ImageSize.height);
    focalLengthY = _CameraIntrinsics->GetFocalLengthY() * factor;
  }

  _ViewAngle = 2 * atan( ( _WindowSize.height / 2 ) / focalLengthY ) * 180 / vnl_math::pi;

应用视角:

d->m_Renderer->GetActiveCamera()->SetViewAngle(viewAngle);

WindowCenter的计算:

  double px = 0;
  double width = 0;

  double py = 0;
  double height = 0;

  if( _ImageSize.width != _WindowSize.width || _ImageSize.height != _WindowSize.height )
  {
    double factor = static_cast<double>(_WindowSize.height)/static_cast<double>(_ImageSize.height);

    px = factor * _CameraIntrinsics->GetPrincipalPointX();
    width = _WindowSize.width;
    int expectedWindowSize = cvRound(factor * static_cast<double>(_ImageSize.width));
    if( expectedWindowSize != _WindowSize.width )
    {
      int diffX = (_WindowSize.width - expectedWindowSize) / 2;
      px = px + diffX;
    }

    py = factor * _CameraIntrinsics->GetPrincipalPointY();
    height = _WindowSize.height;
  }
  else
  {
    px = _CameraIntrinsics->GetPrincipalPointX();
    width = _ImageSize.width;

    py = _CameraIntrinsics->GetPrincipalPointY();
    height = _ImageSize.height;
  }

  double cx = width - px;
  double cy = py;

  _WindowCenter.x = cx / ( ( width-1)/2 ) - 1 ;
  _WindowCenter.y = cy / ( ( height-1)/2 ) - 1;

设置窗口中心:

 d->m_Renderer->GetActiveCamera()->SetWindowCenter(_WindowCenter.x, _WindowCenter.y);

将外在矩阵应用于相机:

// create a scaling matrix (THE CLASS TRANSFORM IS A WRAPPER FOR A 4x4 Matrix, methods should be self-documenting)
d->m_ScaledTransform = Transform::New();
d->m_ScaleMat.set_identity();
d->m_ScaleMat(1,1) = -d->m_ScaleMat(1,1);
d->m_ScaleMat(2,2) = -d->m_ScaleMat(2,2);

// scale the matrix appropriately (m_VnlMat is a VNL 4x4 Matrix)
d->m_VnlMat = d->m_CameraExtrinsicMatrix->GetMatrix();
d->m_VnlMat = d->m_ScaleMat * d->m_VnlMat;
d->m_ScaledTransform->SetMatrix( d->m_VnlMat );

d->m_VnlRotation = d->m_ScaledTransform->GetVnlRotationMatrix();
d->m_VnlRotation.normalize_rows();
d->m_VnlInverseRotation = vnl_matrix_inverse<mitk::ScalarType>( d->m_VnlRotation );

// rotate translation vector by inverse rotation P = P'
d->m_VnlTranslation = d->m_ScaledTransform->GetVnlTranslation();
d->m_VnlTranslation = d->m_VnlInverseRotation * d->m_VnlTranslation;
d->m_VnlTranslation *= -1;  // save -P'

// from here proceed as normal
// focalPoint = P-viewPlaneNormal, viewPlaneNormal is rotation[2]
d->m_ViewPlaneNormal[0] = d->m_VnlRotation(2,0);
d->m_ViewPlaneNormal[1] = d->m_VnlRotation(2,1);
d->m_ViewPlaneNormal[2] = d->m_VnlRotation(2,2);

d->m_vtkCamera->SetPosition(d->m_VnlTranslation[0], d->m_VnlTranslation[1], d->m_VnlTranslation[2]);

d->m_vtkCamera->SetFocalPoint( d->m_VnlTranslation[0] - d->m_ViewPlaneNormal[0],
                               d->m_VnlTranslation[1] - d->m_ViewPlaneNormal[1],
                               d->m_VnlTranslation[2] - d->m_ViewPlaneNormal[2] );
d->m_vtkCamera->SetViewUp( d->m_VnlRotation(1,0), d->m_VnlRotation(1,1), d->m_VnlRotation(1,2) );

最后做一个裁剪范围重置:

d->m_Renderer->ResetCameraClippingRange();

希望这可以帮助.我没有时间解释更多细节.特别是最后一个代码(将extrinsics应用于相机)具有一些与坐标系方向相关的含义.但这对我有用.

最佳迈克尔