Having set up the object detection pipeline, I proceeded to the image processing stage, where the raw visual data and the model’s outputs are transformed into a more artistic representation. My process involved several key steps. First, I applied edge detection algorithms to the video frames. This technique identifies points in a digital image where the brightness changes sharply, effectively outlining the shapes and contours of objects in the scene. Next, I inverted the black and white colour, creating a stark, high-contrast visual style. Finally, I took the bounding boxes generated by the YOLO detection model and redrew them onto this processed image. This layering of machine perception over a stylised version of reality creates a compelling visual dialogue between the actual scene and the AI’s interpretation of it.

With the video pipeline established, I turned my attention to processing the visual data using a combination of powerful tools. The core of this stage is the YOLO (You Only Look Once) object detection model. YOLO is a state-of-the-art, real-time object detection system that identifies and classifies objects in a single pass of an image, making it incredibly fast and efficient. For this project, I am intentionally using the model with the pre-trained COCO (Common Objects in Context) dataset. The COCO dataset is a large-scale collection of images depicting common objects in everyday scenes and is a standard benchmark for training and evaluating computer vision models.

My goal is not to achieve flawless object recognition but rather to play with the inherent “mistakes” and misinterpretations the machine makes. The default COCO dataset is perfectly suited for this, as its generalised training can lead to incorrect predictions when applied to novel or ambiguous scenes. To manipulate the image data, which is essentially a collection of pixels, I am using NumPy (Numerical Python). NumPy is a fundamental library for scientific computing in Python that allows for efficient manipulation of large, multi-dimensional arrays and matrices—the very structure that represents digital images.

What is Object Detection?

Object detection is a field of computer vision and image processing concerned with identifying and locating instances of objects within images and videos. Unlike simple image classification, which assigns a single label to an entire image, object detection models draw bounding boxes around each detected object and assign a class label to it, providing more detailed information about the scene.

What are NumPy and the COCO Dataset?

• NumPy: A Python library that provides support for large, multi-dimensional arrays and matrices, along with a collection of mathematical functions to operate on these arrays. In image processing, an image is treated as a 3D array (height, width, colour channels), making NumPy an indispensable tool for any pixel-level manipulation.
• COCO Dataset: Standing for “Common Objects in Context,” this is a massive dataset designed for object detection, segmentation, and captioning tasks. It contains hundreds of thousands of images with millions of labelled object instances across 80 “thing” categories and 91 “stuff” categories, providing a rich foundation for training computer vision models.

Objects Detectable by the COCO Dataset:

The COCO dataset can identify 80 common object categories, including:

• People: person
• Vehicles: bicycle, car, motorcycle, airplane, bus, train, truck, boat
• Outdoor: traffic light, fire hydrant, stop sign, parking meter, bench
• Animals: bird, cat, dog, horse, sheep, cow, elephant, bear, zebra, giraffe
• Accessories: backpack, umbrella, handbag, tie, suitcase
• Sports: frisbee, skis, snowboard, sports ball, kite, baseball bat, baseball glove, skateboard, surfboard, tennis racket
• Kitchen: bottle, wine glass, cup, fork, knife, spoon, bowl
• Food: banana, apple, sandwich, orange, broccoli, carrot, hot dog, pizza, donut, cake
• Furniture: chair, couch, potted plant, bed, dining table, toilet
• Electronics: tv, laptop, mouse, remote, keyboard, cell phone
• Appliances: microwave, oven, toaster, sink, refrigerator
• Indoor: book, clock, vase, scissors, teddy bear, hair drier, toothbrush

With a large dataset of generated policies, the next step is to import them back into the primary software application that displays the 360-degree video. This integration allows the dynamically generated rules to influence the visual output or behaviour of the system in real-time. My use of the term “policy” is a deliberate nod to its origins in the field of Reinforcement Learning (RL), a concept dating back to the 1990s. In RL, a policy is the strategy an agent employs to make decisions and take actions in its environment. It is the core component that dictates the agent’s behaviour as it learns through trial and error to maximise cumulative reward. By generating policies based on visual input, my system is, in a sense, creating its own world model—a simplified, learned representation of its environment and the relationships within it. This process echoes the fundamental principles of how an AI agent learns to react to and make sense of the real world, a topic I have delved into in more detail in some of my earlier writings.

In a more experimental turn, I developed a separate piece of software to explore the concept of emergent behaviour based on the object detection output. This program uses a Large Language Model (LLM) to generate “policies” when objects from the COCO dataset are detected in close proximity on the screen. The system calculates the normalised distance between the bounding boxes of detected objects. This distance value is then fed to the LLM, which has been prompted to generate a policy or rule based on the perceived danger or interaction potential of the objects being close together. For instance, if a “person” and a “car” are detected very close to each other, the LLM might generate a high-alert policy, whereas a “cup” and a “dining table” would result in a benign, functional policy. This creates a dynamic system where the AI is not just identifying objects, but also creating a narrative or a set of rules about their relationships in the environment.

The initial phase of this project involved tackling the technical groundwork required to process 360-degree video. I began by using OpenCV, a powerful open-source computer vision library, to stitch together the two separate video feeds from my 360-degree camera. OpenCV is an essential tool for real-time image and video processing, providing the necessary functions to merge the hemispheric views into a single, equirectangular frame. After successfully connecting the camera to my computer, I set up a basic Python workspace within my integrated development environment (IDE). The next step was to write a script that could access the camera’s video stream and display it in a new window, confirming that the foundational hardware and software were communicating correctly. This setup provides the visual canvas upon which the subsequent layers of AI-driven interpretation will be built.

I am revisiting a creative process that has captivated my interest for some time: enabling an agent to perceive and learn about its environment through the lens of a computer vision model. In a previous exploration, I experimented with CLIP (Contrastive Language-Image Pre-Training), which led to the whimsical creation of a sphere composed of text, a visual representation of the model’s understanding. This time, however, my focus shifts to the YOLO (You Only Look Once) model. My prior experiences with YOLO, using the default COCO dataset, often yielded amusingly incorrect object detections—a lamp mistaken for a toilet, or a cup identified as a person’s head. Instead of striving for perfect accuracy, I intend to embrace these algorithmic errors. This project will be a playful exploration of the incorrectness and the fascinating illusions generated by an AI model, turning its faults into a source of creative inspiration.

* visualization using CLIP and Blender for artwork “Golem Wander in Crossroads”

Ultralytics YOLO

https://docs.ultralytics.com

Ultralytics YOLO is a family of real-time object detection models renowned for their speed and efficiency. Unlike traditional models that require multiple passes over an image, YOLO processes the entire image in a single pass to identify and locate objects, making it ideal for applications like autonomous driving and video surveillance. The architecture divides an image into a grid, and each grid cell is responsible for predicting bounding boxes and class probabilities for objects centered within it. Over the years, YOLO has evolved through numerous versions, each improving on the speed and accuracy of its predecessors.
(Text from Gemini-2.5-Pro and edited by artist)

CLIP

https://github.com/openai/CLIP

CLIP (Contrastive Language-Image Pre-Training), developed by OpenAI, is a neural network that learns visual concepts from natural language descriptions. It consists of two main components: an image encoder and a text encoder, which are trained jointly on a massive dataset of 400 million image-text pairs from the internet. This allows CLIP to create a shared embedding space where similar images and text descriptions are located close to one another. A key capability of CLIP is “zero-shot” classification, meaning it can classify images into categories it wasn’t explicitly trained on, simply by providing text descriptions of those categories.

(Text from Gemini-2.5-Pro and edited by artist)

COCO

https://cocodataset.org/#home

https://docs.ultralytics.com/datasets/detect/coco

COCO (Common Objects in Context), is a large-scale object detection, segmentation, and captioning dataset. It is designed to encourage research on a wide variety of object categories and is commonly used for benchmarking computer vision models. It is an essential dataset for researchers and developers working on object detection, segmentation, and pose estimation tasks.

(Text from Ultralytics YOLO Docs)