How the Code Works: Yolov8 Extracts Plugin

Overview of Yolov8 Extracts Plugin:

1. Capture Screenshots: The primary functionality of the Yolo Extracts plugin revolves around capturing screenshots. This is a crucial step in creating a labeled dataset for training object detection models. Screenshots are taken from runelite!
2. Generate YOLO-formatted Text Files: Once the screenshots are captured, the plugin processes these images and generates YOLO-formatted text files. YOLO (You Only Look Once) is a popular real-time object detection system that divides the image into a grid and predicts bounding boxes and class probabilities for each grid cell. The generated text files contain information about the bounding boxes and corresponding class labels in a specific format required by YOLO.
3. Machine Learning Data Preparation: The Yolo Extracts plugin significantly simplifies the data preparation process for training object detection models. By creating YOLO-formatted text files, it provides a structured and standardized input for machine learning algorithms. This, in turn, enhances the training efficiency and effectiveness of YOLO-based models.
4. Compatibility with YOLO Versions: One notable feature of the Yolo Extracts plugin is its adaptability to different YOLO versions, including YOLOv8. YOLO is an evolving framework with new versions and improvements being released over time. The plugin’s code is designed to accommodate these changes and seamlessly work with multiple YOLO versions, ensuring that users can benefit from the latest advancements in object detection technology.

Elaboration on YOLOv8 Compatibility:

• Configurability: The Yolo Extracts plugin is configured to handle the specific requirements and nuances of YOLOv8. This includes any changes in the model architecture, input preprocessing, or output formatting introduced in YOLOv8.
• Utilization of YOLOv8 Features: The code within the plugin may take advantage of unique features and optimizations introduced in YOLOv8. This ensures that users can leverage the capabilities of the latest YOLO version without any compatibility issues.
• Future-Proof Design: As YOLO continues to evolve, the Yolo Extracts plugin is designed to be future-proof. This means that it can seamlessly integrate with upcoming YOLO versions, allowing users to stay up-to-date with the latest developments in object detection technology.

In conclusion, the Yolo Extracts plugin serves as an invaluable tool for researchers and developers working with YOLO-based object detection models. Its compatibility with various YOLO versions, including YOLOv8, showcases its flexibility and commitment to keeping pace with advancements in the field of computer vision and machine learning.

Package and Imports

net.runelite.client.plugins.yolo;

import com.google.common.annotations.VisibleForTesting;
import com.google.common.base.Strings;
import com.google.inject.Provides;
import lombok.extern.slf4j.Slf4j;
import net.runelite.api.*;
import net.runelite.api.Point;
import net.runelite.api.coords.LocalPoint;
import net.runelite.api.events.GameTick;
import net.runelite.client.config.ConfigManager;
import net.runelite.client.config.RuneScapeProfileType;
import net.runelite.client.eventbus.Subscribe;
import net.runelite.client.plugins.Plugin;
import net.runelite.client.plugins.PluginDescriptor;
import net.runelite.client.ui.ClientUI;
import net.runelite.client.ui.DrawManager;
import net.runelite.client.util.ImageCapture;
import net.runelite.client.util.Text;

import javax.imageio.ImageIO;
import javax.inject.Inject;
import javax.swing.*;
import java.awt.*;
import java.awt.image.BufferedImage;
import java.io.File;
import java.io.IOException;
import java.lang.reflect.InvocationTargetException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.nio.file.StandardOpenOption;
import java.text.SimpleDateFormat;
import java.util.*;
import java.util.List;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
import java.util.function.Consumer;

import static net.runelite.api.Perspective.localToCanvas;
import static net.runelite.client.RuneLite.SCREENSHOT_DIR;

The code begins with the package declaration and imports necessary classes and libraries for the plugin’s functionality. Notable imports include various classes from the net.runelite.api package for accessing the RuneLite API, as well as utility classes like ImageIO and SwingUtilities for image processing and GUI operations.

Plugin Descriptor

javaCopy code@PluginDescriptor(
    name = "Yolo Extracts",
    description = "takes screenshots and creates YOLO-formatted text files for Machine Learning",
    tags = {"external", "images", "imgur", "integration", "notifications"},
    enabledByDefault = false
)
@Slf4j
public class YoloPlugin extends Plugin

The @PluginDescriptor annotation provides metadata about the plugin, including its name, description, tags, and default enabled status. In this case, the plugin is named “Yolo Extracts” and is responsible for capturing screenshots and generating YOLO-formatted text files for machine learning purposes. The plugin extends the Plugin class and uses the @Slf4j annotation for logging.

Fields and Dependencies

@Inject
private ClientUI clientUi;

@Inject
private Client client;

@Inject
private DrawManager drawManager;

public long currentTime = 0;

@Inject
public YoloConfig config;
@Inject
private ScheduledExecutorService executor;

public int MAX_DISTANCE = 2000;
public int SNAP_TIMER = 3;

public String SAVE_DIRECTORY = null;
@Inject
private ImageCapture imageCapture;
@Provides
private YoloConfig provideConfig(ConfigManager configManager) {
    return configManager.getConfig(YoloConfig.class);
}

The code defines various fields and dependencies required for the plugin’s functionality. These fields are injected using the @Inject annotation and include references to the ClientUI, Client, DrawManager, YoloConfig, `ScheduledExecutorService.

Initialization and Startup

@Override
protected void startUp() throws Exception {
    executor = Executors.newSingleThreadScheduledExecutor();

    String captureDir = config.captureDir();

    if (Strings.isNullOrEmpty(captureDir)) {
        captureDir = SCREENSHOT_DIR;
    }

    SAVE_DIRECTORY = captureDir;

    File file = new File(SAVE_DIRECTORY);

    if (!file.exists()) {
        if (!file.mkdir()) {
            log.warn("Unable to create directory for YoloPlugin screenshots: {}", file);
            return;
        }
    }

    executor.scheduleAtFixedRate(this::captureTick, SNAP_TIMER, SNAP_TIMER, TimeUnit.SECONDS);
}

@Override
protected void shutDown() {
    executor.shutdown();
}

The startUp() method is called when the plugin is started. It initializes the executor with a new single-threaded scheduled executor. The captureDir variable is obtained from the plugin’s configuration, and if it is not specified, the default screenshot directory (SCREENSHOT_DIR) is used.

The SAVE_DIRECTORY field is set to the captureDir value. If the directory doesn’t exist, it is created using the mkdir() method. If the directory creation fails, a warning is logged.

Finally, the captureTick() method is scheduled to run at a fixed rate specified by SNAP_TIMER (3 seconds) using the executor‘s scheduleAtFixedRate() method.

The shutDown() method is called when the plugin is stopped and shuts down the executor.

Screenshot Capture

private void captureTick() {
    if (client.getGameState() != GameState.LOGGED_IN || client.getGameState() == GameState.LOGIN_SCREEN) {
        return;
    }

    if (client.getTickCount() == currentTime) {
        return;
    }

    currentTime = client.getTickCount();

    BufferedImage gameBufferedImage = imageCapture.takeScreenshot(clientUi.isResized());

    if (gameBufferedImage == null) {
        return;
    }

    Point mouseCanvasPosition = client.getMouseCanvasPosition();

    LocalPoint localPoint = client.getLocalPlayer().getLocalLocation();

    int canvasX = localToCanvas(client, localPoint.getX(), localPoint.getY()).getX();
    int canvasY = localToCanvas(client, localPoint.getX(), localPoint.getY()).getY();

    int mouseX = mouseCanvasPosition.getX() - canvasX;
    int mouseY = mouseCanvasPosition.getY() - canvasY;

    String fileName = new SimpleDateFormat("yyyyMMdd_HHmmss").format(new Date());
    String filePath = Paths.get(SAVE_DIRECTORY, fileName + ".png").toString();

    saveScreenshot(gameBufferedImage, filePath);

    String label = String.format("0 %.6f %.6f %.6f %.6f", 0.0, normalizeX(mouseX), normalizeY(mouseY), 0.0);
    saveLabelFile(label, filePath);
}

The captureTick() method is responsible for capturing a screenshot and saving it in the designated directory for yolov8. It performs the following steps:

1. Checks if the client’s game state is in the appropriate state for capturing screenshots.
2. Checks if the current tick count is the same as the previous tick count to avoid capturing duplicate screenshots.
3. Updates the currentTime with the current tick count.
4. Calls the takeScreenshot() method from the imageCapture object to capture the game screen as a BufferedImage.
5. Checks if the captured gameBufferedImage is null (i.e., screenshot capture failed).
6. Retrieves the canvas position of the mouse and the local player’s position on the canvas.
7. Calculates the relative mouse coordinates by subtracting the local player’s canvas position from the mouse’s canvas position.
8. Generates a unique file name using the current timestamp.
9. Constructs the file path by combining the SAVE_DIRECTORY and the generated file name with the “.png” extension.
10. Saves the screenshot to the file path using the saveScreenshot() method.
11. Constructs the label for the screenshot in the YOLO format, representing the object class and normalized coordinates.
12. Saves the label to a label file using the saveLabelFile() method.

Saving Screenshot

private void saveScreenshot(BufferedImage image, String filePath) {
    try {
        ImageIO.write(image, "png", new File(filePath));
    } catch (IOException e) {
        log.warn("Error saving screenshot: {}", e.getMessage());
    }
}

The saveScreenshot() method takes a BufferedImage and a file path as input. It uses the ImageIO.write() method to save the image as a PNG file to the specified file path. If any IO exception occurs during the saving process, a warning message is logged.

Saving Label File

private void saveLabelFile(String label, String filePath) {
    String labelFilePath = filePath.replace(".png", ".txt");

    try (PrintWriter writer = new PrintWriter(labelFilePath)) {
        writer.write(label);
    } catch (FileNotFoundException e) {
        log.warn("Error saving label file: {}", e.getMessage());
    }
}

The saveLabelFile() method takes a label string and a file path as input. It generates the label file path by replacing the ".png" extension with ".txt". Then, it creates a PrintWriter to write the label string to the label file. If a FileNotFoundException occurs during the process, a warning message is logged.

This method assumes that the label file format is a simple text file containing the label string.

That concludes the code snippet related to capturing and saving screenshots in the YOLOv8 plugin.

Annoting the files

The code represents a Java method called annoteFiles that is responsible for annotating files with bounding box information using XML format. Let’s break down the code and understand its functionality step by step:

public void annoteFiles(String datetime, List yoloName, List yoloMinx, List yoloMaxx, List yoloMiny, List yoloMaxy) {

This method takes several parameters: datetime (a string representing the current date and time), and five List objects (yoloName, yoloMinx, yoloMaxx, yoloMiny, yoloMaxy) that store information related to the bounding boxes.

System.out.println("annote files:" + SAVE_DIRECTORY + datetime + ".png");

This line simply outputs a message to the console indicating the file that is being annotated.

String width = String.valueOf(clientUi.getWidth());
String height = String.valueOf(clientUi.getHeight());
String depth = "3";
String file_name = datetime + ".png";
String folder_name = SAVE_DIRECTORY;
String file_path = SAVE_DIRECTORY;

These lines initialize several variables with relevant information for the XML annotation. It retrieves the width and height of the clientUi (presumably an interface) and assigns them to the width and height variables. The depth variable is set to 3, indicating a color image with RGB channels. The file_name variable is formed by concatenating the datetime parameter with the file extension “.png”. The folder_name and file_path variables are set to the value of SAVE_DIRECTORY, which is presumably a directory path.

try {
	File myObj = new File(SAVE_DIRECTORY + datetime + ".xml");
	Path output = Paths.get(SAVE_DIRECTORY + datetime + ".xml");

These lines create a File object (myObj) and a Path object (output) representing the XML file that will be created to store the annotations. The file name is formed by concatenating SAVE_DIRECTORY, datetime, and the “.xml” extension.

try {
	Files.write(output, "<annotation>\n".getBytes(), StandardOpenOption.CREATE, StandardOpenOption.APPEND);
	// Writing XML elements and their corresponding values
} catch (IOException e) {
	System.out.println("Annote files, an error occurred.");
}

This code block begins the XML writing process. The opening <annotation> tag is written to the XML file. Then, several XML elements such as <folder>, <filename>, <path>, <size>, and <object> are written to the file, along with their corresponding values. These values are obtained from the variables initialized earlier. The code uses the Files.write() method to append the XML elements and values to the file.

for (int i = 0; i < endLoop; i++) {
	Files.write(output, "\t<object>\n".getBytes(), StandardOpenOption.CREATE, StandardOpenOption.APPEND);
	add_objects(output, yoloName, yoloMinx, yoloMaxx, yoloMiny, yoloMaxy, i);
	Files.write(output, "\t</object>\n".getBytes(), StandardOpenOption.CREATE, StandardOpenOption.APPEND);
}

This loop iterates over the provided bounding box information (yoloName, yoloMinx, yoloMaxx, yoloMiny, yoloMaxy) and writes the corresponding <object> elements to the XML file for each entry. Inside the loop, the method add_objects is called to append the XML elements related to the bounding box information. The add_objects method writes the <name>, <pose>, <truncated>, <difficult>, and <bndbox> elements along with their values to the XML file.

Files.write(output, "</annotation>\n".getBytes(), StandardOpenOption.CREATE, StandardOpenOption.APPEND);

After the loop completes, the closing </annotation> tag is written to the XML file.

if (myObj.createNewFile()) {
	System.out.println("File created: " + myObj.getName());
} else {
	System.out.println("File already exists for XML.");
}

This code block checks if the XML file already exists. If it doesn’t, a new file is created using createNewFile() and a success message is printed. Otherwise, a message indicating that the file already exists is printed.

The annoteFiles method is responsible for creating an XML file and writing annotation information for each bounding box. The add_objects method is called to write the specific XML elements related to the bounding boxes. This code is useful for generating XML annotations for image datasets with bounding box information.

Annotating Objects in XML: Exploring the add_objects Method

When working with image datasets, it is often necessary to annotate objects within the images with bounding box information. This helps in various computer vision tasks such as object detection, object recognition, and localization. In this blog post, we will explore the add_objects method, which plays a crucial role in appending object annotations to an XML file. Let’s dive into the code and understand its functionality.

Understanding the add_objects Method: The add_objects method is responsible for writing XML elements related to individual objects and their bounding box information. Let’s break down the code and explore its components step by step.

public void add_objects(Path output, List yoloName, List yoloMinx, List yoloMaxx, List yoloMiny, List yoloMaxy, int x) {
    try {
        String name = (String) yoloName.get(x);
        String minx = (String) yoloMinx.get(x);
        String maxx = (String) yoloMaxx.get(x);
        String miny = (String) yoloMiny.get(x);
        String maxy = (String) yoloMaxy.get(x);

        // Writing the <name> element
        Files.write(output, "\t\t<name>".getBytes(), StandardOpenOption.CREATE, StandardOpenOption.APPEND);
        Files.write(output, name.getBytes(), StandardOpenOption.CREATE, StandardOpenOption.APPEND);
        Files.write(output, "</name>\n".getBytes(), StandardOpenOption.CREATE, StandardOpenOption.APPEND);

        // Writing the <pose> element
        Files.write(output, "\t\t<pose>".getBytes(), StandardOpenOption.CREATE, StandardOpenOption.APPEND);
        Files.write(output, "Unspecified".getBytes(), StandardOpenOption.CREATE, StandardOpenOption.APPEND);
        Files.write(output, "</pose>\n".getBytes(), StandardOpenOption.CREATE, StandardOpenOption.APPEND);

        // Writing the <truncated> element
        Files.write(output, "\t\t<truncated>".getBytes(), StandardOpenOption.CREATE, StandardOpenOption.APPEND);
        Files.write(output, "0".getBytes(), StandardOpenOption.CREATE, StandardOpenOption.APPEND);
        Files.write(output, "</truncated>\n".getBytes(), StandardOpenOption.CREATE, StandardOpenOption.APPEND);

        // Writing the <difficult> element
        Files.write(output, "\t\t<difficult>".getBytes(), StandardOpenOption.CREATE, StandardOpenOption.APPEND);
        Files.write(output, "0".getBytes(), StandardOpenOption.CREATE, StandardOpenOption.APPEND);
        Files.write(output, "</difficult>\n".getBytes(), StandardOpenOption.CREATE, StandardOpenOption.APPEND);

        // Writing the <bndbox> element and its child elements
        Files.write(output, "\t\t<bndbox>\n".getBytes(), StandardOpenOption.CREATE, StandardOpenOption.APPEND);
        Files.write(output, "\t\t\t<xmin>".getBytes(), StandardOpenOption.CREATE, StandardOpenOption.APPEND);
        Files.write(output, minx.getBytes(), StandardOpenOption.CREATE, StandardOpenOption.APPEND);
        Files.write(output, "</xmin>\n".getBytes(), StandardOpenOption.CREATE, StandardOpenOption.APPEND);
        Files.write(output, "\t\t\t<ymin>".getBytes(), StandardOpenOption.CREATE, StandardOpenOption.APPEND);
        Files.write(output, miny.getBytes(), StandardOpenOption.CREATE, StandardOpenOption.APPEND);
        Files.write(output, "</ymin>\n".getBytes(), StandardOpenOption.CREATE, StandardOpenOption.APPEND);
        Files.write(output, "\t\t\t<xmax>".getBytes(), StandardOpenOption.CREATE, StandardOpenOption.APPEND);
        Files.write(output, maxx.getBytes(), StandardOpenOption.CREATE, StandardOpenOption.APPEND);
        Files.write(output, "</xmax>\n".getBytes(), StandardOpenOption.CREATE, StandardOpenOption.APPEND);
        Files.write(output, "\t\t\t<ymax>".getBytes(), StandardOpenOption.CREATE, StandardOpenOption.APPEND);
        Files.write(output, maxy.getBytes(), StandardOpenOption.CREATE, StandardOpenOption.APPEND);
        Files.write(output, "</ymax>\n".getBytes(), StandardOpenOption.CREATE, StandardOpenOption.APPEND);
        Files.write(output, "\t\t</bndbox>\n".getBytes(), StandardOpenOption.CREATE, StandardOpenOption.APPEND);

    } catch (IOException e) {
        System.out.println("Adding objects to XML: An error occurred.");
    }
}

Exploring the Code:

1. The add_objects method takes several parameters: output (the XML file path), yoloName (a list containing the object names), yoloMinx (a list of minimum x-coordinates of bounding boxes), yoloMaxx (a list of maximum x-coordinates), yoloMiny (a list of minimum y-coordinates), yoloMaxy (a list of maximum y-coordinates), and x (the index of the current object being processed).
2. The method starts by retrieving the relevant information (object name, bounding box coordinates) from the respective lists based on the given index.
3. The method then uses the Files.write() method to append XML elements and their values to the output file. Let’s understand each step in detail:
   • <name> Element: The object name is written between the <name> opening and closing tags. This provides a unique identifier for the object being annotated.
   • <pose> Element: The pose is set to “Unspecified” using the <pose> opening and closing tags. It indicates that the pose of the object is not considered in the annotation.c.
   • <truncated> Element: The truncated value is set to “0” using the <truncated> opening and closing tags. It signifies that the object is not truncated.d. <difficult> Element: The difficult value is set to “0” using the <difficult> opening and closing tags. It indicates that the object is not considered difficult to detect or classify.
   • <bndbox> Element: The <bndbox> opening tag is written to mark the beginning of the bounding box information. This tag encloses the four child elements: <xmin>, <ymin>, <xmax>, and <ymax>.
   • <xmin>, <ymin>, <xmax>, and <ymax> Elements: The corresponding coordinates (minimum x, minimum y, maximum x, maximum y) of the bounding box are written using their respective opening and closing tags. These values define the position and size of the bounding box around the object.g.
   • </bndbox> Element: The </bndbox> closing tag is written to mark the end of the bounding box information.
4. If any exception occurs during the file writing process, an error message is displayed.

Click this guide to using the YOLO plugin: How to Use the Yolo Extracts Plugin for RuneLite and How to Configure the YOLO Plugin in Java

Conclusion: The add_objects method plays a crucial role in appending XML elements related to individual objects and their bounding box information. By using this method, we can efficiently annotate objects within an image dataset and generate XML annotations required for various computer vision tasks. Understanding this code is essential for working with image datasets and performing tasks like object detection and localization.