SONY TOF

1. Overview

SONY TOF Camera is applicated in Xvisio SeerSense™ DS80 series products. Refer to the corresponding datasheet of other products to confirm the type of TOF camer. SONY TOF camera also uses the iTOF algorithm, with a wavelength of 940nm.

2. Features

2.1 Work Mode

SONY TOF Camera supports multiple working modes to cope with different host processing capabilities and application scenarios:

Work mode definition:

enum class SonyTofLibMode { IQMIX_DF ,IQMIX_SF, LABELIZE_DF ,LABELIZE_SF ,M2MIX_DF ,M2MIX_SF };

2.2 Modulation Frequency

Support single frequency 60MHz or dual frequency 20MHz/120MHz. All the working modes support these two modulation frequencies. Generally, it is recommended to use single frequency mode within 2 meters of working distance. Dual frequency mode is recommended for working distance greater than 2 meters. For dual frequency, the distance is longer and the accuracy is higher. The data bandwidth and load of single frequency are lower. Please refer to the product datasheet for specific parameters.
Definition of modulation frequency:

enum class SonyTofLibMode { IQMIX_DF ,IQMIX_SF, LABELIZE_DF ,LABELIZE_SF ,M2MIX_DF ,M2MIX_SF };

Modulation frequency mode and working mode share the same type.

2.3 Resolution

Support two resolutions:

• VGA：640*480
• QVGA：320*240

All the work mode support these two resolutions.

Resolution definition:

enum class Resolution{ Unknown = -1,VGA = 0 ,QVGA ,HQVGA};

Note: SONY TOF doesn't support HQVGA.

2.4 Frame Rate and Data Bandwidth

TOF Frame rate can be set from 5FPS to 30FPS. Due to the load limit of computing power bandwidth under different working modes, the setting takes effect only when the corresponding frame rate can meet the setting requirements. Please see the table below for the corresponding frame rate:

Users can choose different modes and settings according to usage scenarios and requirements.
Frame rate definition:

enum class Framerate{ FPS_5 ,FPS_10 ,FPS_15 ,FPS_20 ,FPS_25 ,FPS_30 };

2.5 Set API

Xvisio SDK provides an API which can set "work mode","modulation frequency","resolution" and "frame rate" simultaneously:

device->tofCamera()->setSonyTofSetting(xv::TofCamera::SonyTofLibMode::IQMIX_DF,
            xv::TofCamera::Resolution::VGA,
            xv::TofCamera::Framerate::FPS_30);

Refer to demo-api.cpp（case 11）for detail code information. demo-api

2.6 Filter Setting File

Xvisio SDK provides a filter setting file that can replace the current TOF algorithm library. If you want to use this function, you need to have a certain understanding of the filter format and functions of the TOF algorithm library.
Interface prototype：
void setFilterFile(std::string filePath)
Call example:

device->tofCamera()->setSonyTofSettingsetFilterFile("filterfile.bin");

3 Depth Data and IR Data

SONY TOF depth data format supports 16-bit depth data with a unit of millimeter which also support IR gray data.

3.1 DepthImage Definition

/**
 * @brief An image provided by a TOF camera.
 * @note  There are two manufacturers of TOF camera, Pmd and sony.
 *        Pmd TOF depth type is Depth_32，sony TOF depth type is Depth_16.
 *        Cloud type just use for Pmd point cloud,the coordinate system of the point cloud is the camera coordinate system, and the data unit is meters.
 *        Length, width and depth are in meters use Pmd TOF.
 *        Length, width and depth are in metmillimeterers use sony TOF. 
 */
struct DepthImage {
    enum class Type { Depth_16 = 0, Depth_32, IR, Cloud, Raw, Eeprom, IQ };
    Type type = Type::Depth_32;
    std::size_t width = 0; //!< width of the image (in pixel)
    std::size_t height = 0; //!< height of the image (in pixel)
    double  confidence = 0.0; //!< confidence of depth [0.0,1.0]
    std::shared_ptr<const std::uint8_t> data = nullptr; //! image of depth
    unsigned int dataSize = 0;
    double hostTimestamp = std::numeric_limits<double>::infinity(); //!< host timestamp of the physical measurement (in second based on the `std::chrono::steady_clock`).
    std::int64_t edgeTimestampUs = (std::numeric_limits<std::int64_t>::min)(); //!< timestamp of the physical measurement (in microsecond based on edge clock).
    /**
     * @brief Convert to a #xv::RgbImage
     */
    RgbImage toRgb() const;
};

Notes:

1. The value of “type” is “Depth_16” or “IR”.
2. “width” and “height” is corresponds with TOF resolution.
3. “data” indicates depth image data with a unit of millimeter. The bit width of each pixel is 16 bits. The corresponding conversion is carried out according to the application scenario.
4. “data size” indicates "width" * "height" * 2 (bitwidth of 16bits).
5. “confidence” in valid in IQ mode.
6. “hostTimestamp” is synchronously converted to the timestamp of the same baseline on the host side by the timestamp of the SDK. The unit is seconds and the accuracy reach to microseconds.
7. “edgeTimestampUs” is the Xvisio device sampling timestamp with a unit of microseconds
8. “toRgb()” is temporarily invalid, reserved.

3.2 Get DepthImage

Get DepthImage by register callback way. The interface call process is as follows:

1. Register tofCamera callback,
2. Call start() to start TOF streaming,
3. Trigger callback, and do data processing in callback,
4. Call stop() after deregistering callback to stop TOF Streaming.
   Refer to demo-api.cpp（case 11）for detail code. demo-api

4 Point Cloud Data

Xvisio SDK provides API which can convert depth data to point cloud data format.

4.1 PointCloud Definition

/**
 * @brief A point cloud of 3D points.
 */
struct PointCloud {
    double hostTimestamp = std::numeric_limits<double>::infinity(); //!< host timestamp of ? (in second based on the `std::chrono::steady_clock`).
    std::int64_t edgeTimestampUs = (std::numeric_limits<std::int64_t>::min)(); //!< timestamp of ? (in microsecond based on edge clock).
    std::vector<Vector3f> points;
};

Notes:

1. After the timestamp synchronization of SDK, “hostTimestamp” is converted to the timestamp of the same baseline on the host side. The unit is second and the accuracy can reach to microsecond.
2. “edgeTimestampUs” indicates Xvisio device sampling timestamp (unit:s).
3. “points” indicates point cloud data. Each pixel is composed of three float types,which representing the depth data of x, y, z axis (unit:mm).

4.2 Get PointCloud

To get Pointcloud, firstly you should obtain the DepthImage, and then call the interface for conversion. The process is as follows:

1. Register callback of tofCamera,
2. Call start() to start TOF streaming,
3. Trigger callback, and use API to do point cloud convert processing in callback:
   device->tofCamera()->depthImageToPointCloud(tofDepthImage)
4. Call stop() after deregistering callback to stop TOF Streaming.
   Refer to demo-api.cpp（case 11）for detail code. demo-api

5 Code

• Define Xvisio Device
  std::shared_ptr<xv::Device> device = nullptr;
• Read device_list, set timeout value to 10 seconds(recommonds value which can be shorten.)
  auto device_list = xv::getDevices(10., "");
• Judge whether the obtained device is empty. It fails if it is empty.

  if (device_list.empty())`
  {
          LOG(LOG_Function,"initDevice faiiled:Timeout for device detection.");
          return false;
  }
  
  

• Get device （only for single equipment, please refer to the corresponding document description for multiple equipments.）
  device = device_list.begin()->second;
• Set "Work mode" to "IQ single frequency", set "frame rate" to 30fps, set "resolution" to "VGA resolution":

  bool ret = m_xvDevice->tofCamera()->setSonyTofSetting(xv::TofCamera::SonyTofLibMode::IQMIX_SF,
                                          xv::TofCamera::Resolution::VGA,
                                          xv::TofCamera::Framerate::FPS_30);
      if(!ret)
      {
          std::cout<<"set TOF Setting failed"<<std::endl;
      }
  
  

• Register callback. Get depth data or IR data and point cloud data in the callback definition.

  int tofId = -1;
  tofId = device->tofCamera()->registerCallback([&](xv::DepthImage const & tofDepthImage){
          if (tof.type == xv::DepthImage::Type::Depth_16 ) {
              //Operation on tofDepthImage
              //...
  
              //point cloud process
              if(enableDevMap["tof_point_cloud"])
              {
                  auto points = device->tofCamera()->depthImageToPointCloud(tofDepthImage)->points;
          
                  for (auto iter = points.begin(); iter != points.end();iter++)
                  {
                      auto point = *iter;
                      printf(buff, "x=%f ,y=%f ,z=%f\n", point[0], point[1], point[2]);
                      //...
                  }
          
              }
          }
          else if(tof.type == xv::DepthImage::Type::IR )
          {
              //Operation on tofDepthImage
              //...
          }
      });
  
  

• Start TOF

  //start tof
  device->tofCamera()->start();
  

• Stop TOF
  The tofId used here is the value assigned when registering callback.

  device->tofCamera()->unregisterCallback(tofId);
  device->tofCamera()->stop();
  

6 Notes:

1. The current peak of the device has certain requirements for the USB power supply load capacity of the host. After enabling TOF, the TOF camera has an infrared projector, which will periodically emit infrared waves and bring periodic load peaks. Refer to Xvisio Device and User Guide for detail information.
2. The depth data formats and units of PMD TOF and SONY TOF are different. PMD TOF is 32bits with a unit of meter. SONY TOF is 16bits with a unit of millimeter.

←  Xvisio SDK Depth
←  Xvisio Features and Quick Start
←  Xvisio SDK Documentation Home Page