Vision sensor filter composition
A vision sensor normally produces two images at each simulation pass: a color image and a depth map. Those two images can be programmatically inspected by retrieving them through appropriate API function calls, then iterating over each individual pixel or depth map value. While this approach allows a maximum of flexibility, it is however troublesome and impractical. Instead, it is much more convenient (and fast!) to use the built-in filtering and triggering capabilities. Indeed, each vision sensor has a filter associated, that can be composed in a very flexible way by combining several components. Following figure illustrates a simple filter that inverts colors:
[Vision sensor filter with 3 components]
A component can perform 4 basic operations:
Following figure illustrates the various types of buffers a component can access:
[Vision sensor buffers and operations between buffers]
While the input image and input depth image are volatile buffers (i.e. normally automatically overwritten with new data at each simulation pass), the work image, buffer1 image and buffer2 image are persistent buffers (i.e. their content is not modified unless a component operates on them, e.g. persistent buffers can be used to compare vision sensor data from one simulation pass to the next simulation pass).
A vision sensor is triggered if at least one component of its filter activates the trigger. The API function sim.handleVisionSensor (or sim.readVisionSensor) returns the sensor's trigger state, and a series of values or packets. The first packet may contain different data depending on the vision sensor settings:
If additional filter components return values, then they will be appended as packets to the first packet. See the function's API description for more details.
V-REP has more than 30 built-in filter components that can be combined as needed. In addition, new filter components can be developed through plugins. When a filter component from a plugin is used, then you should always distribute the created scene with the plugin, or you should make sure to check whether the plugin is present with the sim.getModuleName API command in a child script (or C-application).