PMD TOF

1. Overview

TOF will be widely used with Xvisio SeerSense™ GR45 and SeerLens™ One glass Pro modules. Refer to the corresponding datasheet of other modules to confirm the type of TOF camera. The PMD TOF camera also uses the iTOF algorithm, with a wavelength of 940nm.

2. Features

2.1 Work Mode

Xvisio SDK provides three modes to apdat to different appplication. System will automatically adapt to the corresponding frequency mode, frame rate and exposure time after setting the mode:

Work mode definition:

enum class DistanceMode { Short = 0, Middle, Long };

Set API examples:

device->tofCamera()->setDistanceMode(TofCamera::DistanceMode::Short);

2.2 Data Stream Mode

Xvisio SDK provides a setting interface to set data stream mode. Different mode corresponds to different data stream types or combinations.

Definition:

enum class StreamMode { DepthOnly = 0, CloudOnly, DepthAndCloud, None, CloudOnLeftHandSlam};

Set API Examples:

device->tofCamera()->setStreamMode(TofCamera::StreamMode::DepthAndCloud);

2.3 Modulation Frequency

Xvisio SDK does't open the setting interface of modulation frequency" setting interface for PMD TOF camera. The corresponding single and dual frequency mode can only be selected by setting the working mode.

2.4 Resolution

Xvisio SDK doesn't open the resolution setting for PMD TOF camera, the dafault resolution is 224*172.

2.5 Frame Rate

The frame rates can be set with a range of 5-30FPS. Please check the table for the exposure time corresponding to different frame rates:

Users can choose different modes and settings according to usage scenarios and nrequirements.
Set API Examples:

device->tofCamera()->setFramerate(30.0f);

The frame rate value of parameter transmission is float type, with a range of 5-30.

2.6 Bandwidth

The data volume of PMD TOF is relatively small. Here are different data type bandwidths under 30fps:

2.7 Enable IR Data Stream

PMD TOF can enable or disable IR data stream output. It is default that IR data stream can't be output.
It is set by HID command. IR enable command: “0x02,0x10,0xf5,0x02,0x01”，IR disable command: “0x02,0x10,0xf5,0x02,0x00” .
Set API Examples:

std::vector<unsigned char> result(63);
bool bOK = device->hidWriteAndRead({0x02,0x10,0xf5,0x02,0x01}, result);
if(bOK)
    std::cout << "Enable IR successfully" << std::endl;
else
    std::cout << "Enable IR failed" << std::endl;

3 Depth Data/IR Data/Cloud Data

PMD TOF depth data and point cloud data are 32-bit depth data with a unit of meter which also support IR gray data.

3.1 DepthImage Definition

/**
 * @brief An image provided by a TOF camera.
 * @note  There are two manufacturers of TOF camera, Pmd and sony.
 *        Pmd TOF depth type is Depth_32，sony TOF depth type is Depth_16.
 *        Cloud type just use for Pmd point cloud,the coordinate system of the point cloud is the camera coordinate system, and the data unit is meters.
 *        Length, width and depth are in meters use Pmd TOF.
 *        Length, width and depth are in metmillimeterers use sony TOF. 
 */
struct DepthImage {
    enum class Type { Depth_16 = 0, Depth_32, IR, Cloud, Raw, Eeprom, IQ };
    Type type = Type::Depth_32;
    std::size_t width = 0; //!< width of the image (in pixel)
    std::size_t height = 0; //!< height of the image (in pixel)
    double  confidence = 0.0; //!< confidence of depth [0.0,1.0]
    std::shared_ptr<const std::uint8_t> data = nullptr; //! image of depth
    unsigned int dataSize = 0;
    double hostTimestamp = std::numeric_limits<double>::infinity(); //!< host timestamp of the physical measurement (in second based on the `std::chrono::steady_clock`).
    std::int64_t edgeTimestampUs = (std::numeric_limits<std::int64_t>::min)(); //!< timestamp of the physical measurement (in microsecond based on edge clock).
    /**
     * @brief Convert to a #xv::RgbImage
     */
    RgbImage toRgb() const;
};

Notes:

1. The value of “type” is “Depth_32” or “IR” or “Cloud”.
2. “width” and “height” is corresponds with TOF resolution.
3. "type" determines the type of "data":
   When "type" is “Depth_32”,“data” is depth data with a unit of meter. 32-bit width per pixel and "float" type.
   When "type" is “Cloud”,“data” indicates piont cloud data. The unit is meter. Each pixel is composed of three float types,which representing the depth data of X, y, z axis (unit:m).
   When "type" is “IR”,“data” indicates IR data which means the gray value. Each pixel is represented by a 16 bit width. In fact, only the lower 8 bits are valid, and the upper 8 bits are 0.
4. “data size” indicates "width" * "height" * "bitwidth". The bitwidth of “Depth_32” is 32bits.The bitwidth of “Cloud” is 128bits. The bitwidth of “IR” is 16bits.
5. “confidence” is temporarily invalid, reserved.
6. “hostTimestamp” is synchronously converted to the timestamp of the same baseline on the host side by the timestamp of the SDK. The unit is seconds and the accuracy reach to microseconds.
7. “edgeTimestampUs” is the Xvisio device sampling timestamp with a unit of microseconds.
8. “toRgb()” is temporarily invalid, reserved.

3.2 Get DepthImage

Get DepthImage by register callback way. The interface call process is as follows:

1. Register tofCamera callback,
2. Call start() to start TOF streaming,
3. Trigger callback, and do data processing in callback,
4. Call stop() after deregistering callback to stop TOF Streaming.
   Refer to demo-api.cpp（case 11）for detail code. demo-api 。

4 Convert Point Cloud Data

Xvisio SDK provides API which can convert depth data to point cloud data format.
Section 3 introduces that if point cloud data is directly obtained in DepthImage, this method is reported on the Xvisio device side. It also supports the API introduced in this section to convert depth data into point cloud data format.
Both of the methods can be used, and users can choose one of them.

4.1 PointCloud Definition

/**
 * @brief A point cloud of 3D points.
 */
struct PointCloud {
    double hostTimestamp = std::numeric_limits<double>::infinity(); //!< host timestamp of ? (in second based on the `std::chrono::steady_clock`).
    std::int64_t edgeTimestampUs = (std::numeric_limits<std::int64_t>::min)(); //!< timestamp of ? (in microsecond based on edge clock).
    std::vector<Vector3f> points;
};

Notes：

1. After the timestamp synchronization of SDK, “hostTimestamp” is converted to the timestamp of the same baseline on the host side. The unit is second and the accuracy can reach to microsecond.
2. “edgeTimestampUs” indicates Xvisio device sampling timestamp (unit:s).
3. “points” indicates point cloud data. Each pixel is composed of three float types,which representing the depth data of X, y, z axis (unit:m).

4.2 Convert PointCloud

To convert Pointcloud, firstly you should obtain the DepthImage, and then call the interface for conversion. The process is as follows:

1. Register callback of Camera,
2. Call start() to start TOF streaming,
3. Trigger callback, and use API to do point cloud convert processing in callback:
   device->tofCamera()->depthImageToPointCloud(tofDepthImage)
4. Call stop() after deregistering callback to stop TOF Streaming.
   Refer to demo-api.cpp（case 11）for detail code. demo-api

5 Code

• Define Xvisio Device
  std::shared_ptr<xv::Device> device = nullptr;

• Read device_list, set timeout value to 10 seconds(recommonds value which can be shorten.)
  auto device_list = xv::getDevices(10., "");

• Judge whether the obtained device is empty. It fails if it is empty.

  if (device_list.empty())`
  {
          LOG(LOG_Function,"initDevice faiiled:Timeout for device detection.");
          return false;
  }
  
  

• Get device（only for single equipment, please refer to the corresponding document description for multiple equipments.）
  device = device_list.begin()->second;

• Register callback. Get depth data or IR data and point cloud data in the callback definition.

  int tofId = -1;
  tofId = device->tofCamera()->registerCallback([&](xv::DepthImage const & tofDepthImage){
          if (tof.type == xv::DepthImage::Type::Depth_16 ) {
              //Operation on tofDepthImage
              //...
  
              //point cloud process
              if(enableDevMap["tof_point_cloud"])
              {
                  auto points = device->tofCamera()->depthImageToPointCloud(tofDepthImage)->points;
          
                  for (auto iter = points.begin(); iter != points.end();iter++)
                  {
                      auto point = *iter;
                      printf(buff, "x=%f ,y=%f ,z=%f\n", point[0], point[1], point[2]);
                      //...
                  }
          
              }
          }
          else if(tof.type == xv::DepthImage::Type::IR )
          {
              //Operation on tofDepthImage
              //...
          }
           else if(tof.type == xv::DepthImage::Type::Cloud )
          {
              //Operation on tofDepthImage
              //...
          }
      });
  

• Start TOF

  //start tof
  device->tofCamera()->start();
  

• Stop TOF
  The tofId used here is the value assigned when registering callback.

  device->tofCamera()->unregisterCallback(tofId);
  device->tofCamera()->stop();
  

6 Notes

1. The current peak of the device has certain requirements for the USB power supply load capacity of the host. After enabling TOF, the TOF camera has an infrared projector, which will periodically emit infrared waves and bring periodic load peaks. Refer to Xvisio Device and User Guide for detail information.
2. The depth data formats and units of PMD TOF and SONY TOF are different. PMD TOF is 32bits with a unit of meter. SONY TOF is 16bits with a unit of millimeter.

←  Xvisio SDK Depth
←  Xvisio Features and Quick Start
←  Xvisio SDK Documentation Home Page