1. VEX Library
  2. V5
  3. AI Vision Sensor
  4. Coding with the AI Vision Sensor

Coding with the AI Vision Sensor in VEXcode V5 Blocks

Make sure you have Color Signatures and Color Codes configured with your AI Vision Sensor so they can be used with your blocks. To learn more about how to configure them, you can read the articles below:

The AI Vision Sensor can also detect AI Classifcations and AprilTags. To learn how to enable these detection modes, go here:

To learn more detail about these individual Blocks and how to use them in VEXcode, go to the API site.

Take Snapshot

Diagram illustrating the V5 AI Vision Sensor, showcasing its components and functionality in a robotics context, including camera placement and sensor features.

The Take Snapshot block takes a picture of what the AI Vision Sensor is currently seeing and pulls data from that snapshot that can then be used in a project. When a snapshot is taken, you need to specify what type of object the AI Vision Sensor should collect data of:

  • Color Signature
  • Color Code
  • AI Classifications
  • AprilTags

Taking a snapshot will create an array of all of the detected objects that you specified. For instance, if you wanted to detect a "Red" Color Signature, and the AI Vision Sensor detected 3 different red objects, data from all three would be put in the array.

For more information on how to specify between different objects, go to the "Set Object Item" section in this article.

A light blue coding block with the command to take a snapshot of an object or color. There are two dropdown options: one labeled AIVision2 and the other labeled Blue. The block is designed for use in a block-based coding environment, where it captures a snapshot from an AI Vision sensor and tracks an object or color defined as Blue. The block has slight curves, typical of coding interfaces that use modular blocks.

In this example, it will only detect objects that match its configured “Blue” Color Signature and nothing else.

Data Taken From a Snapshot

Keep in mind that the AI Vision Sensor will use its last taken snapshot for any Blocks that come after. To make sure you're always getting the most up-to-date information from your AI Vision Sensor, retake your snapshot every time you want to pull data from it. 

Width and Height

This is the width or height of the detected object in pixels.

The image shows a blue Buckyball with a white square outline tracking it. The top left corner has a label indicating it is a blue object, with coordinates X:176, Y:117, and dimensions W:80, H:78. Red arrows highlight the width and height of the object.

The width and height measurements help identify different objects. For example, a Buckyball will have a larger height than a Ring.

Two blue cubic objects being tracked by a visual recognition system. The upper cube has a white outline with a label indicating its position as X:215, Y:70 and dimensions W:73, H:84. The lower cube has a similar white outline with the label displaying X:188, Y:184 and dimensions W:144, H:113. Each cube has a centered white cross, likely indicating the focal point for tracking. The labels highlight the measurements and tracking data for each object.

Width and height also indicate an object's distance from the AI Vision Sensor. Smaller measurements usually mean the object is farther away, while larger measurements suggest it's closer.

The program starts with the block when started, followed by a forever loop. Inside the loop, the program takes a snapshot using the AI Vision sensor (AIVision1) to detect a blue object. If the object exists, the program checks if the width of the object is less than 250 pixels. If true, the robot drives forward; otherwise, it stops driving. The blocks are stacked together, indicating the flow of the program in a modular coding environment.

In this example, the width of the object is used for navigation. The robot will approach the object until the width has reached a specific size before stopping.

CenterX and Center Y

This is the center coordinates of the detected object in pixels.

A blue Buckyball being tracked by a computer vision system. The object is outlined with a white square, and inside the outline is a smaller red square surrounding a centered white cross. In the top-left corner of the image, a label indicates the object is blue, with coordinates X:176, Y:117, and dimensions W:80, H:78.

CenterX and CenterY coordinates help with navigation and positioning. The AI Vision Sensor has a resolution of 320 x 240 pixels.

Two blue cubic objects tracked by a vision system. The upper object is labeled with coordinates X:215, Y:70, and dimensions W:73, H:84, with a white outline and a centered white cross. The lower object is labeled with coordinates X:188, Y:184, and dimensions W:144, H:113, also outlined in white with a centered white cross.

You can see that an object closer to the AI Vision Sensor will have a lower CenterY coordinate than an object that is farther away.

A block-based coding sequence starting with when started followed by a forever loop. Inside the loop, the program takes a snapshot using AIVision1 to detect a blue object. If an object exists, the program turns until the object is centered in the AI Vision sensor's view. The object is considered centered if its centerX value is between 150 and 170. If the object is not centered, the robot turns right; otherwise, it stops driving. The blocks indicate the flow and logic of the visual program.

In this example, because the center of the AI Vision Sensor's view is (160, 120), the robot will turn right until a detected object's centerX coordinate is greater than 150 pixels, but less than 170 pixels.

Angle

Diagram illustrating the features and components of the V5 AI Vision Sensor, including sensor layout, connectivity options, and usage in robotics applications.

Angle is a property only available for Color Codes and AprilTags. This represents if the detected Color Code or AprilTag is orientated differently.

A stack of two cubes, one green on top and one blue on the bottom, being tracked by a vision system. A white outline surrounds both cubes, with a white cross centered on the green cube. The label at the bottom of the image displays Green_Blue A:87°, indicating the detected colors and an angle measurement. Below that, the coordinates are listed as X:117, Y:186, with dimensions W:137, H:172, representing the position and size of the stacked cubes in the frame.

You can see if the robot is orientated differently in relation to the Color Code or AprilTag and make navigation decisions according to that.

Two cubes, one green and one blue, placed side by side and tracked by a vision system. A white outline surrounds both cubes with a white cross at the center. The top-left label indicates Green_Blue A:0°, referencing the detected colors and an angle measurement. Below that, the coordinates are shown as X:150, Y:102, with dimensions W:179, H:109, representing the position and size of the cubes within the frame.

For instance, if a Color Code isn't detected at a proper angle, then the object it represents may not be able to be picked up properly by the robot.

OriginX and OriginY

OriginX and OriginY is the coordinate at the top-left corner of the detected object in pixels.

A blue Buckyball being tracked by a vision system. A white outline surrounds the object, with a centered white cross inside the outline. The top-left label indicates the object's color as blue, along with coordinates X:176, Y:117, and dimensions W:80, H:78. A small red square highlights the object's top-left corner.

OriginX and OriginY coordinates help with navigation and positioning. By combining this coordinate with the object's Width and Height, you can determine the size of the object's bounding box. This can help with tracking moving objects or navigating between objects.

A block-based coding sequence beginning with when started followed by a forever loop. Inside the loop, the program takes a snapshot using AIVision1 to detect a blue object. If the object exists, the program will draw a rectangle on the Brain screen based on the object's position and size. The rectangle is drawn using the object's originX, originY, width, and height values, which are provided by the AIVision1 sensor. This program visually tracks and highlights the detected object on the Brain screen.

In this example, a rectangle will be drawn on the Brain using the exact coordinates of its origin, width, and height.

tagID

The tagID is only available for AprilTags. This is the ID number for the specified AprilTag.

Three square cards with AprilTags being tracked by a vision system. Each card is labeled with an ID number and associated tracking data. The card on the left is labeled ID:0, showing coordinates A:350°, X:110, Y:96, W:41, H:41. The middle card, labeled ID:3, has coordinates A:1°, X:187, Y:180, W:57, H:57. The card on the right is labeled ID:9, with coordinates A:3°, X:237, Y:89, W:38, H:38. Each card has a white outline, and the system is tracking their positions and orientations.

Identifying specific AprilTags allows for selective navigation. You can program your robot to move towards certain tags while ignoring others, effectively using them as signposts for automated navigation.

Score

The score property is used when detecting AI Classifications with the AI Vision Sensor.

The image shows four objects being tracked by a vision system: two balls and two rings. The red ball is labeled with coordinates X:122, Y:84, W:67, H:66, and a score of 99%. The blue ball has X:228, Y:86, W:70, H:68, with a score of 99%. The green ring has coordinates X:109, Y:186, W:98, H:92, and a score of 99%. The red ring is labeled X:259, Y:187, W:89, H:91, with a score of 99%. Each object is outlined in white, indicating tracking accuracy.

The confidence score indicates how certain the AI Vision Sensor is about its detection. In this image, it's 99% confident in identifying these four objects' AI Classifications. You can use this score to ensure your robot only focuses on highly confident detections.

Set Object Item

When an object is detected by the AI Vision Sensor, it's put into an array. By default, the AI Vision Sensor will pull data from the first object in the array, or the object with the index of 1. If your AI Vision Sensor has only detected one object, then that object will  be selected by default.

When your AI Vision Sensor has detected multiple objects at once, however, you'll need to use the Set Object Item block to specify which object you want to pull data from.

A light blue coding block. It contains a command to set the object item for AIVision1 to 1. This block is part of a block-based coding environment, typically used to define which object or item the AI Vision sensor should focus on or track. The shape of the block has slight curves, fitting into the modular nature of the visual coding platform.

When multiple objects are detected by the AI Vision Sensor, they are arranged in the array by largest to smallest. That means that the largest detected object will always be set to object index 1, and the smallest object will always be set to the highest number.

The AI Vision Utility interface with two blue cubes detected on the left side, each marked with their X and Y coordinates and dimensions. The system is connected, and AprilTags are toggled on, while AI Classification is off. On the right, the Blue color settings are displayed with adjustable hue and saturation ranges, set at 22 and 0.34, respectively. There is an option to add or set color and freeze video. The firmware is up to date, running version 1.0.0.b16, and a close button is available at the bottom.

In this example, two objects have been detected with the Color Signature "Blue". They both will be put in the array when the Take Snapshot block is used.

The AI Vision Utility interface, tracking two blue cubes labeled with their X, Y, and dimension data. The left cube has coordinates X:127, Y:179, and dimensions W:136, H:123, while the right cube has coordinates X:233, Y:74, and dimensions W:78, H:87. The system is connected, AprilTags is turned on, and AI Classification is off. The blue color settings have a hue range of 22 and a saturation of 0.34. A Freeze Video button and firmware information (version 1.0.0.b16) are displayed at the bottom.

Here, the object in the front would become object index 1, since it is the largest object, and the smallest object would become object index 2.

Object Exists

Before pulling any data from a snapshot, it's important to always check to make sure the AI Vision Sensor has detected any objects from that snapshot first. This is where the Object Exists block comes into play.

A light blue hexagonal coding block with the text AIVision1 object exists? This block is part of a block-based coding environment, typically used to check if an object is detected by the AI Vision sensor labeled as AIVision1. The block is designed to fit within a modular coding structure, with the slight curves and shape characteristic of such environments.

This block will return a True or False value on whether or not the last taken snapshot has any objects detected in it.

This block should always be used to ensure you're not trying to pull any data from a potentially empty snapshot.

A block-based coding sequence that starts with when started followed by a forever loop. Inside the loop, the AI Vision sensor (AIVision2) takes a snapshot to detect the color Blue. If an object with the Blue visual signature is detected, the robot will drive forward. If no object is detected, the robot will stop driving. The blocks are stacked to represent the conditional logic of the program, where the presence of a detected object controls the robot's movement.

For instance, here the robot will be constantly taking snapshots with the AI Vision Sensor. If it identifies any object with the “Blue” Color Signature, it will drive forward.


If any snapshot does not have the “Blue” Color Signature, the robot will stop moving.

Object Count

A light blue, rounded coding block labeled AIVision1 object count. This block is used in a block-based coding environment to retrieve the number of objects detected by the AI Vision sensor labeled as AIVision1. The block fits within a modular structure, commonly used in visual programming interfaces for robotics or vision systems.

Using the Object count block will allow you to see how many objects of a specific Color Signature the AI Vision Sensor can see in its last snapshot. 

The AI Vision Utility interface with two blue cubes detected on the left. The system is connected, with AprilTags enabled and AI Classification turned off. The interface displays settings for detecting the color blue, with hue set to 22 and saturation to 0.34. Buttons for freezing the video and adding or setting the color are present. Firmware is indicated as up to date, running version 1.0.0.b16. There is also a button to disconnect the connection or close the utility.

Here, we see the AI Vision Sensor has the configured Color Signature “Blue”, and is detecting two objects.

A block-based coding sequence starting with when started, followed by a forever loop. Inside the loop, the program takes a snapshot using AIVision2 to detect the blue visual signature. It clears and resets the console before checking if any blue objects are detected. If a blue object exists, the object count is printed to the console. The program then waits for two seconds before repeating the process. The blocks visually represent a continuous check for blue objects, displaying the results in the console.Diagram illustrating the V5 AI Vision Sensor, showcasing its components and functionality within the V5 robotics system, highlighting key features for enhanced visual processing and object detection.

In this code, the AI Vision Sensor would take a snapshot and print “2” on the VEXcode console, since it only detects two “Blue” Color Signatures.

Object

A dropdown menu for selecting object properties related to AIVision1. The selected option is width, and the menu lists other options including height, centerX, centerY, angle, originX, originY, tagID, and score. These options allow the user to retrieve specific data points from the object detected by the AI Vision sensor, providing flexibility for tracking or measuring various attributes of the object.

The Object block allows you to report the property of your specified object. This lets you use any of the available data pulled from the most recently taken snapshot.

Object properties that can be pulled from taken snapshots are:

  • width
  • height
  • centerX
  • centerY
  • angle
  • originX
  • originY
  • tagID
  • score

Read the "Data Taken from Snapshot" section of this article for more information on these properties.

Detected AprilTag is

A light blue hexagonal block from a coding interface. It contains the command to check whether the detected AprilTag by AIVision1 matches the value 1. This block is used in a block-based coding environment and is designed to evaluate whether a specific AprilTag is present. The shape and structure allow it to fit within other logic blocks, typically used in visual programming for robotics or AI vision tasks.

The Detected AprilTag is block is only available when the AprilTag Detection Mode is turned on.

This block will report True or False depending on if the specified object is a certain AprilTag.

Three AprilTags being tracked by a vision system. Each tag has an ID and associated coordinates. The left tag is labeled ID:0, with coordinates X:110, Y:96, W:41, H:41. The center tag is labeled ID:3, with coordinates X:187, Y:180, W:57, H:57. The right tag is labeled ID:9, with coordinates X:237, Y:89, W:38, H:38.

When multiple AprilTags are detected in a single snapshot, they are arranged in the array based on their identified ID, not by size.

In this image, three AprilTags are detected with IDs 0, 3, and 9. They will be organized in ascending order of their ID in the array. The object at index 1 would correspond to the AprilTag with ID 0, at index 2 to the AprilTag with ID 3, and at index 3 to the AprilTag with ID 9.

For more information on what AprilTags are and how to enable their detection with the AI Vision Sensor, read this article.

AI Classification is

A light blue hexagonal block from a block-based coding environment. The block contains the command AIVision1 AI classification is BlueBall? This block checks whether the AI Vision sensor (AIVision1) has classified the detected object as a BlueBall. The block features dropdown options, allowing users to select different objects or AI classifications for tracking and decision-making purposes in a visual programming interface.

The AI Classification is block is only available when the AI Classification Detection Mode is turned on.

 

This block will report True or False depending on if the specified object is a certain AI Classification.

What AI Classifications can be detected by the AI Vision Sensor varies depending on what model you are using. For more information on what AI Classifications are available and how to enable their detection with the AI Vision Sensor, read this article.

For more information, help, and tips, check out the many resources at VEX Professional Development Plus

Last Updated: