With the video pipeline established, I turned my attention to processing the visual data using a combination of powerful tools. The core of this stage is the YOLO (You Only Look Once) object detection model. YOLO is a state-of-the-art, real-time object detection system that identifies and classifies objects in a single pass of an image, making it incredibly fast and efficient. For this project, I am intentionally using the model with the pre-trained COCO (Common Objects in Context) dataset. The COCO dataset is a large-scale collection of images depicting common objects in everyday scenes and is a standard benchmark for training and evaluating computer vision models.

My goal is not to achieve flawless object recognition but rather to play with the inherent “mistakes” and misinterpretations the machine makes. The default COCO dataset is perfectly suited for this, as its generalised training can lead to incorrect predictions when applied to novel or ambiguous scenes. To manipulate the image data, which is essentially a collection of pixels, I am using NumPy (Numerical Python). NumPy is a fundamental library for scientific computing in Python that allows for efficient manipulation of large, multi-dimensional arrays and matrices—the very structure that represents digital images.

What is Object Detection?

Object detection is a field of computer vision and image processing concerned with identifying and locating instances of objects within images and videos. Unlike simple image classification, which assigns a single label to an entire image, object detection models draw bounding boxes around each detected object and assign a class label to it, providing more detailed information about the scene.

What are NumPy and the COCO Dataset?

• NumPy: A Python library that provides support for large, multi-dimensional arrays and matrices, along with a collection of mathematical functions to operate on these arrays. In image processing, an image is treated as a 3D array (height, width, colour channels), making NumPy an indispensable tool for any pixel-level manipulation.
• COCO Dataset: Standing for “Common Objects in Context,” this is a massive dataset designed for object detection, segmentation, and captioning tasks. It contains hundreds of thousands of images with millions of labelled object instances across 80 “thing” categories and 91 “stuff” categories, providing a rich foundation for training computer vision models.

Objects Detectable by the COCO Dataset:

The COCO dataset can identify 80 common object categories, including:

• People: person
• Vehicles: bicycle, car, motorcycle, airplane, bus, train, truck, boat
• Outdoor: traffic light, fire hydrant, stop sign, parking meter, bench
• Animals: bird, cat, dog, horse, sheep, cow, elephant, bear, zebra, giraffe
• Accessories: backpack, umbrella, handbag, tie, suitcase
• Sports: frisbee, skis, snowboard, sports ball, kite, baseball bat, baseball glove, skateboard, surfboard, tennis racket
• Kitchen: bottle, wine glass, cup, fork, knife, spoon, bowl
• Food: banana, apple, sandwich, orange, broccoli, carrot, hot dog, pizza, donut, cake
• Furniture: chair, couch, potted plant, bed, dining table, toilet
• Electronics: tv, laptop, mouse, remote, keyboard, cell phone
• Appliances: microwave, oven, toaster, sink, refrigerator
• Indoor: book, clock, vase, scissors, teddy bear, hair drier, toothbrush