Ethan White

Interested in using computer vision to help understand, manage, and
conserve nature? We hiring for a new position to join our team working
on models, software, and community outreach in our existing projects
using airborne imagery to monitor and understand natural systems at
large scales.

The Weecology lab at the University of Florida has an opening for a
computer vision developer to join our team. We use airborne imagery
across research programs to monitor trees and wildlife and develop open
source tools for the biological monitoring community. The position will
contribute to research projects, Python package development, and
community outreach. Areas of computer vision research include
fine-grained classification, active learning and sensor fusion from data
collected from UAVs, crewed aircraft, and (potentially) satellites. Our
open source projects include
[DeepForest](https://deepforest.readthedocs.io/), the [MillionTrees
benchmark](https://milliontrees.idtrees.org/), as well as upcoming
efforts for a wide array of challenging biodiversity applications.

Ideal candidates will work effectively with a history of responsibility
and attention to detail. Independence and a willingness to take
ownership over projects is key. Some experience with Python, open-source
software development practices, and deep learning frameworks is
preferred, but there is also ample room for skill building and
mentorship within the lab composed of research scientists, software
engineers and biologists. Our lab has a proven track record in
developing machine learning software and datasets for ecology. Your work
will be valued and used by scientists around the world contributing to
biodiversity monitoring with a global reach.

We are committed to supporting and training a diverse scientific
workforce. Current and former group members encompass a variety of
racial and ethnic backgrounds from the U.S. and other countries, members
of the LGBTQIA+ community, military veterans, and people with
disabilities. We work hard to create a supportive and inclusive lab
environment and expect all members of Weecology to abide by [the lab
code of conduct](https://www.weecology.org/lab-wiki/code-of-conduct/). 

Interested applicants should contact Dr. Glenda Yenni (gmyenni@ufl.edu)
by November 30th. Fully remote work is possible for this position.
Anticipated salary is \$60,000 -- \$70,000 annually (depending on
experience) including full benefits and there is potential flexibility
in salary and work arrangements for highly qualified applicants. In your
email, please include as a single attached file: 1) why you are
interested in this position (300 words or less) and 2) a resume or CV
including descriptions and links to prior work and/or research
experience and the responsibilities, skills, and duties involved in
those positions.


Interested in using computer vision to help understand, manage, and conserve nature? We hiring for a new position to join our team working on models, software, and community outreach in our existing projects using airborne imagery to monitor and understand natural systems at large scales. The Weecology lab at the University of Florida has an opening for a computer vision developer to join our team.

We’re hiring a computer vision developer!

Yunzhong Zhang

**Efficient creation of a stoplight report with data dashboard images**

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*SgLFlOJN2wLSMtg0" />
<figcaption>Image generated by DALL.E</figcaption>
</figure>

### Author:

- Yunzhong Zhang (ORCID:
  [0009--0002--8177--419X](https://orcid.org/0009-0002-8177-419X))

Comparing data dashboards is crucial for understanding trends and
performance differences. Traditionally, this task required manual
effort, which was slow and sometimes inaccurate. Now, thanks to OpenAI's
GPT-4 with Vision (GPT-4V), we are able to automate and improve this
process. This article introduces how to use GPT-4V to compare two data
dashboards quickly and accurately.

Traditionally, to compare two data dashboards and generate a stoplight
report that highlights the differences between them, the process
requires the machine to possess the capability to recognise and compare
text within the images. This involves at least three steps:

1.  **Text Extraction:** Initially, data displayed on both dashboards
    must be extracted. This is achieved through Optical Character
    Recognition (OCR), which extracts text from the images.
2.  **Data Extraction (Feature Selection):** The next step is to
    identify the key metrics and features presented in both dashboards,
    focusing on the essential data for comparison.
3.  **Comparison and Stoplight Report Generation:** With the data
    extracted from the two dashboards, it's possible to compare the
    differences and generate a stoplight report based on these findings.

However, customising the code structure at each step is often necessary.
This is particularly true for comparing charts, graphs, or other visual
elements within the dashboards, which requires an understanding of the
types of visualisations used (e.g., bar charts, line graphs, heatmaps)
and the data they represent.

ChatGPT makes life easy. ChatGPT, developed by OpenAI, can handle a wide
range of natural language processing (NLP) tasks. In September 2023,
OpenAI released the multimodal model GPT-4 with Vision (GPT-4V), which
marries the language processing prowess of GPT-4 with image analysis
capabilities, offering a comprehensive multimodal AI experience. With
GPT-4V, the task of comparing data dashboards is greatly streamlined. By
crafting an appropriate prompt and supplying the dashboard images to
GPT-4V, the model can analyse the data within the images and provide a
detailed stoplight report, making the comparison process more efficient
and user-friendly.

### What is GPT-4V and Stoplight report?

GPT-4 with vision enables users to instruct
[GPT-4](https://openai.com/research/gpt-4) (a Large Multimodal Model) to
analyse image inputs and text prompts provided by the user. With the
ability to interpret and analyse data presented in visual formats, the
GPT-4V is able to monitor the performance, identify trends and patterns,
and enable Real-time Analysis.

A stoplight report uses the colours of a traffic light (red, yellow,
green) to indicate the status or importance of the indicators or
metrics. The colours represent:

- **Green**: Indicates that the metric or indicator is on track,
  performing well, or meeting its target. It signals that no immediate
  action is required.
- **Yellow**: Suggests caution. The metric is somewhat off target, or
  there may be potential issues that need attention. It serves as a
  warning that while not critical, some action may be needed to ensure
  the target is met or to improve performance.
- **Red**: Signifies that the metric or indicator is off target,
  performing poorly, or in a critical state. It alerts users that
  immediate action is necessary to address issues or to get back
  on track.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*lmV7qfDW92qVeSxj" />
<figcaption>Pipeline of using GPT-4V to generate stoplight.</figcaption>
</figure>

### Example: ARC-ROR Data Dashboard

The following screenshots are from the same dashboard on different
dates. It basically presents the status of the Australian research
council and registry. We will use these two screenshots as examples to
explain how to use LLM to generate stoplight reports for dashboards. We
assume that,

- Percentage of organisations that participated in ARC is very
  important, it is essential to monitor its change
- Number of project, Number of organisation, Number of organisation with
  ROR are less important
- Number of organisation with ROR, Number of organisations by country,
  it is no problem if the changes are not significant

Based upon on these assumption, we are able to define the traffic light:

- **Red**: Percentage of organisations that participated in ARC,
- **Yellow**: Number of project, Number of organisation, Number of
  organisation with ROR
- **Green**: Number of organisation with ROR, Number of organisations
  by country

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*fjIRxhkkeKR4UHa8" />
<figcaption>ARC-ROR 08-Jan-2024</figcaption>
</figure>

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*JkZ-cqF1Y1M10daF" />
<figcaption>ARC-ROR 13-Feb-2024</figcaption>
</figure>

#### Step 0: OpenAi API Preparation

We need OpenAi Api Key to access the GPT-4V.

    import base64
    import requests
    import json
    import configparser
    import os

    # OpenAI API Key
    current_path = os.path.dirname(os.path.realpath(__file__))
    config = configparser.ConfigParser()
    config.read(current_path + '/config.ini')
    api_key = config['DEFAULT']['api_key']

#### Step 1: Define the stoplights

We will assign specific colours to the indices, highlighting their
significance on the dashboards. We also establish thresholds for the
degree of change. If one index' s change is significant, the stoplight
colour of this index would be changed to another colour to indicate the
changes. what

    """
    Red:Percentage of organizations that participated in ARC,
    Yellow:Number of project, Number of organization, Number of organization with ROR
    Green:Number of organization with ROR, Number of organizations by country,

    Note that also take the significant change of the data into consideration,
    - If the change is under 1%, the stoplight should be green
    - If the change is between 1% and 5%, the stoplight should be yellow
    - If the change is over 5%, the stoplight should be red

    The priority of the stoplight is:
    Change > Definition of stoplight for the index

    That is to say, assume an index is defined as green, but it increases 8%(>5%), then the stoplight of this index should be red. 
    """

#### Step 2: Design the response template

We create a response template to provide the format and content range
for the response from chatGPT4-V.

    The response should strictly follow the following json template, also make sure the response is in json format.:
    {
       "Percentage of organisations that participated in ARC": {
           "First image": "20%",
           "Second image": "25%",
           "Stoplight": "red"
       },
       "Number of project": {
           "First image": "100",
           "Second image": "150",
           "Stoplight": "yellow"
       },
       "Number of organisation": {
           "First image": "50",
           "Second image": "75",
           "Stoplight": "yellow"
       },
       "Number of organisation with ROR": {
           "First image": "30",
           "Second image": "45",
           "Stoplight": "green"
       },
       "Number of organisations by country": [
           {
               "Country name": "Australia",
               "First image": "10%",
               "Second image": "12%",
               "Stoplight": "green"
           },
           {
               "Country name": "United States",
               "First image": "15%",
               "Second image": "18%",
               "Stoplight": "green"
           }
       ],
    }

#### Step 3: Encode the images

Base64 encoding ensures that your image data can be safely embedded into
text-based API requests, which is especially important for text-based
data exchange formats like JSON. Also, by directly sending image data,
it avoids security and privacy risks associated with image hosting
issues or URL access permissions.

    def encode_image(image_path):
     with open(image_path, "rb") as image_file:
       return base64.b64encode(image_file.read()).decode('utf-8')


    # Path to the images
    image_path = "./NL0008/nl0008-1.1.0.png"
    image_path_2 = "./NL0008/nl0008-1.0.8.png"
    # Getting the base64 string
    base64_image = encode_image(image_path)
    base64_image_2 = encode_image(image_path_2)

#### Step 4: Call GPT-4V

We pass the images of data dashboards, system prompt which includes
response template and the definition of stoplight, and user prompts to
OpenAI' s gpt-4-vision-preview model. Check
[here](https://platform.openai.com/docs/guides/vision) to know more
about
[gpt-4-vision-preview](https://platform.openai.com/docs/guides/vision).

    headers = {
     "Content-Type": "application/json",
     "Authorization": f"Bearer {api_key}"
    }

    payload = {
      "model": "gpt-4-vision-preview",
      "messages": [
        {
          "role": "system", "content": system_message,
        },
        {
          "role": "user",
          "content": [
            # system prompt
            {
              "type": "text",
              "text": "Can you help me compare the two images and tell me what's difference?"
            },
            {
              "type": "image_url",
              "image_url": {
                "url": f"data:image/jpeg;base64,{base64_image}"
              }
            },
            {
              "type": "image_url",
              "image_url": {
                "url": f"data:image/jpeg;base64,{base64_image_2}"
               }
            }
          ]
        }
      ],
      "max_tokens": 500,
    }
    response = requests.post("https://api.openai.com/v1/chat/completions", headers=headers, json=payload)

#### Output

OpenAI API gives the response following the design of the template and
colour of the stoplight is well assigned, where we can clearly see the
importance and the significant change of the metrics.

    {
        "Percentage of organisations that participated in ARC": {
            "First image": "20%",
            "Second image": "19%",
            "Stoplight": "green"
        },
        "Number of project": {
            "First image": "32238",
            "Second image": "30088",
            "Stoplight": "red"
        },
        "Number of organisation": {
            "First image": "14722",
            "Second image": "14152",
            "Stoplight": "yellow"
        },
        "Number of organisation with ROR": {
            "First image": "2950",
            "Second image": "2663",
            "Stoplight": "red"
        },
        "Number of organisations by country": [
            {
                "Country name": "Australia",
                "First image": "20.8%",
                "Second image": "30.0%",
                "Stoplight": "red"
            },
            {
                "Country name": "United States",
                "First image": "18.9%",
                "Second image": "19.5%",
                "Stoplight": "green"
            }
        ]
    }

We also convert data from JSON format to CSV format for enhanced
visualisation.

    import csv
    import json
    with open('data.json', 'r', encoding='utf-8') as json_file:
       data = json.load(json_file)
    if isinstance(data, str):
       data = json.loads(data)

    # csv writer
    with open('structured_data.csv', 'w', newline='', encoding='utf-8') as file:
       writer = csv.writer(file)

       # column name
       writer.writerow(['Category', 'First image', 'Second image', 'Stoplight'])
       for category, details in data.items():
           if isinstance(details, list):
               for detail in details:
                   for key in detail:
                       if key not in ['First image', 'Second image', 'Stoplight']:
                           key_location = key
                   writer.writerow([
                       f"{category} - {detail[key_location]}",
                       detail['First image'],
                       detail['Second image'],
                       detail['Stoplight']
                   ])
           else:
               writer.writerow([
                   category,
                   details['First image'],
                   details['Second image'],
                   details['Stoplight']
               ])

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*YPYkqTRxnOpcSNQk" />
<figcaption>Table output for the information comparison.</figcaption>
</figure>

### Conclusion

GPT-4 Vision excels at interpreting and analysing data displayed in
visual formats, including graphs, charts, and various data
visualisations. It is highly effective in tracking and identifying
differences between two data dashboards, providing valuable insights
into changes and trends.

#### **Sources**

- <https://platform.openai.com/docs/guides/vision>
- <https://gptpluginz.com/gpt-4-vision-api/>

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f32c25ef8d62){width="1"
height="1"}


<strong>
 Efficient creation of a stoplight report with data dashboard images
</strong>
Author:  Yunzhong Zhang (ORCID: 0009–0002–8177–419X)  Comparing data dashboards is crucial for understanding trends and performance differences. Traditionally, this task required manual effort, which was slow and sometimes inaccurate. Now, thanks to OpenAI’s GPT-4 with Vision (GPT-4V), we are able to automate and improve this process.

Rogue Scholar Posts

We’re hiring a computer vision developer!

How to use GPT-4V For Stoplight Report