Computer Vision Algorithms: Decoding the Visual World

Computer vision is a technology powered by artificial intelligence that enables robots, computers, and other machines to process visual information and react accordingly. Computer vision works using cameras and lenses that capture images and AI algorithms that instruct the machine how to interact with the image. These computer vision algorithms enable AI models to detect and classify objects within images, track objects through video or sequential images, and generate unique visuals.

These functions allow professionals in many industries to use computer vision for better information gathering and productivity. Computer vision can be used for business intelligence, more accurate medical diagnostics, agriculture, home protection monitoring, and self-driving vehicles.

Explore how computer vision works and some core computer vision algorithms that empower this technology, including image classification, object detection, object tracking, edge detection, segmentation, and image generation.

Core computer vision algorithms and how they work

Computer vision algorithms enable AI to process, understand, classify, and manipulate images. This technology works similarly to how humans see and understand visual information. Just as you have learned to process visual data throughout your life, computer vision uses training data to provide the AI model with a foundation of visual information. When you see something new, your brain compares it to other things you’ve seen in the past to try to classify or make sense of the unfamiliar. Similarly, computer vision algorithms draw on the patterns noticed in training data to make sense of a new image.

The algorithm is the instruction you provide to the AI model that gives it functionality. You can use different types of algorithms to enable different functions of an AI model, such as detecting or classifying objects or generating new images. You can think about computer vision algorithms in two ways: by their function and by the structure or architecture of the model.

Explore core computer vision algorithms by function and learn the algorithm structure you might use to accomplish each task, including image classification, object detection, object tracking, edge detection, segmentation, and image generation. 

Image classification

Image classification is a method of sorting images by class or a primary characteristic that describes the image. For example, an image classification program might sort pictures of apples, bananas, and oranges. You can also use image classification to predict if an image belongs to a certain class, such as determining whether something is or is not a watermelon. You can use this technology in many ways, including automatically categorizing uploaded images or enabling a camera to focus on faces before taking a picture.

Possible types of algorithms for image classification: Convolutional neural networks, deep convolutional neural networks, logistic regression, support vector machines, and k-nearest neighbor

Object detection

Object detection is an algorithm that digs a little deeper by assessing whether objects meet quality standards or sorting them within a class by appearance. For example, a farmer might use object detection to look for signs of illness in his livestock, or factory workers might look for defective products on the assembly line. Home security systems also use object detection to spot threats like unfamiliar faces.

Possible types of algorithms for object detection: Region-based convolutional neural networks, Single Shot Detector, YOLO (You Only Look Once), RetinaNet, and Feature Pyramid Network

Object tracking

Object tracking algorithms allow computers to track objects moving through a visual field, such as on a video feed or in photos taken sequentially. This technology can be used in many ways, such as monitoring traffic, medical imaging, and autonomous cars, which need to track other moving vehicles, pedestrians, and unmoving objects to avoid collisions.

Possible types of algorithms for object tracking: Single shot detector (SSD), Dense Optical Flow, Kalman Filtering, and Mean Shift

Feature/edge detection

Another type of computer vision algorithm used for image processing and classification is feature detection. Detecting features such as edges, objects, subjects, and background of an image is a critical task that a computer must complete before it can classify the image or make decisions about it. Detecting the edge of an image is particularly important because it can create an outline of the objects in the image to help the computer understand how to process it. After you use an algorithm for edge detection, you can analyze the photo in other ways, such as feature detection, line detection, or edge thinning.

Possible types of algorithms for feature and edge detection: Canny, Roberts, Gaussian, and fuzzy logic

Instance or semantic segmentation

Instance or semantic segmentation are methods for understanding the boundaries of different objects within images and categorizing them individually. For example, instance segmentation may recognize a picture with an adult woman and a newborn baby as two separate subjects within the picture. Semantic segmentation would allow the AI model to understand them as two separate categories of people.

Possible types of algorithms for instance segmentation: Mask region-based convolutional neural networks, Intersection over Union (IoU), and Average Precision (AP)

Image generation

An image generation algorithm can create unique images based on natural language prompts. To accomplish this, these AI models use deep learning and a gamified competition between two neural networks that together form a generative adversarial network (GAN). One part of the GAN AI works as the generator and creates original images based on training data. The other part of the AI model works as a discriminator to spot the differences between the AI-generated image and real images from the training data. The two personas compete until the generator wins the game, that is, until the generated image fools the discriminator. This winning version is sent as the output. Other potential algorithms that you can use for image generation include neural style transfers and stable diffusion models.

Possible types of algorithms for image generation: Generative adversarial networks, neural style transfers, diffusion models

Computer vision careers

If you want to use computer vision algorithms to build AI models that solve real-world problems, consider a career in computer vision. Three potential options include computer vision engineer, robotics engineer, and virtual reality developer.

Computer vision engineer

Average annual pay in the US: $115,137 [1]

Job outlook (projected growth from 2023 to 2033): 26 percent [2]

As a computer vision engineer, you will use computer vision algorithms to build machine learning solutions for your employer or clients who hire your team. You will develop, test, and train computer vision algorithms, create user guides, or train staff on how to use your AI solutions. 

Robotics engineer

Average annual pay in the US: $107,053 [3]

Job outlook (projected growth from 2023 to 2033): 11 percent [4]

As a robotics engineer, you will design, develop, test, and train robots for many industries, such as manufacturing, automotive, health care, national defense, and utilities. Although your exact work will depend on your industry and project, you will likely work with a team of professionals to create, troubleshoot, and implement robots and automated systems. 

Learn more about computer vision algorithms on Coursera.

Computer vision algorithms enable robots and machines to detect and process visual information and respond accordingly. If you’d like to learn more about computer vision algorithms or to start a career in a related field, you can begin today on Coursera. For example, you could enroll in First Principles of Computer Vision Specialization offered by Columbia University.