Australian National University

Qingqin Fang

Transformative Advances in Language Models through External Knowledge Integration

Author: Qingqin Fang (

 ORCID:

0009–0003–5348–4264)

 Introduction

In the dynamic field of natural language processing, the integration of external knowledge has emerged as a pivotal strategy for enhancing the performance of language models.

Deeper understanding of In-Context Retrieval-Augmented Language Models

Wendi Fan

Let’s explore GR-2 together. Author Wendi Fan (

 ORCID: 0000–0003–0284–9166

) Introduction GR-2 is a cutting-edge generative model designed for versatile and generalisable robot manipulation, developed by the Robotics Research Team at ByteDance Research. It represents a significant step forward in robotics, enabling robots to execute a wide range of manipulation tasks in various environments.

GR-2: A Generative Video-Language-Action Model for Robot Manipulation

Vaibhav Khobragade

From Naive to Modular: Tracing the Evolution of Retrieval-Augmented Generation

Author · Vaibhav Khobragade (

 ORCID:

0009–0009–8807–5982) Introduction Large Language Models (LLMs) have achieved remarkable success.

Three Paradigms of RAG

Wenyi Pi

Exploring innovative Strategies in Combating Misinformation with Enhanced Multimodal Understanding Author Wenyi Pi (

 ORCID

: 0009–0002–2884–2771) Introduction Misinformation refers to false or inaccurate information that is often given to someone in a deliberate attempt to make them believe something that is not true. This has a significantly negative impact on public health, political stability and social trust and harmony.

Multimodal Large Language Models for Misinformation Detection and Reasoning

Amanda Kau

Techniques to integrate Knowledge Graphs into Language Models

 Author

Amanda Kau

 (ORCID:


 0009–0004–4949–9284


 )

Introduction Both knowledge graphs (KGs) and pre-trained language models (PLMs) have gained popularity due to their ability to comprehend world knowledge and their broad applicability.

A Deep Dive Into Knowledge Graph Enhanced Pre-trained Language Models

Dhruv Gupta

Understanding the Power and Applications of Natural Language Processing

 Author

Dhruv Gupta (

 ORCID:

0009–0004–7109–5403) Introduction We are living in the era of generative AI. In an era where you can ask AI models almost anything, they will most certainly have an answer to the query. With the increased computational power and the amount of textual data, these models are bound to improve their performance.

NLU vs. NLG: Unveiling the Two Sides of Natural Language Processing

Zijian Yang

Le Chat by Mistral: Access ChatGPT-Like Features, Image Generation, Canvas, Artifact, and More for Free


 Discover Le Chat: Free ChatGPT Alternatives with Canvas, Internet Search, and Flux.1 Image Generation


 Author

Zijian Yang (

 ORCID:

0009–0006–8301–7634)

 Introduction

Imagine accessing premium ChatGPT features — Canvas, internet search, and PDF

Le Chat by Mistral: Access ChatGPT-Like Features, Image Generation, Canvas, Artifact, and More for…

Enhancing Data Interactivity with LLMs and Neo4j Knowledge Graphs Author Wenyi Pi (

 ORCID

: 0009–0002–2884–2771) Introduction Since OpenAI launched ChatGPT, a

 large language model (LLM)

based chatbot, in 2023, it has set off a technological wave.

The integration of large language models (LLMs) with Neo4j-based knowledge graphs

Xuzeng He

Latest findings in multiple research directions for tackling reasoning and common sense challenges Author: Xuzeng He (

 ORCID:

0009–0005–7317–7426) Knowledge Graphs, such as Wikidata, contain rich relational information between entities and have been widely used as a structured format for storing and representing relational information.

Large Language Models and Knowledge Graphs: Ways to combine them

Swinburne University of Technology

Nakul Nambiar

Aishwarya Nambissan

A novel approach to improving the efficiency of text search in graph databases utilizing Neo4j, OpenAI, and Typesense. Authors  Nakul Nambiar (ORCID: 0009–0009–9720–9233) Aishwarya Nambissan (ORCID: 0009–0003–3823–6609)  The ability to use cutting-edge tools and frameworks is essential for staying ahead in the ever-changing field of technology.

Typesense and Neo4j in a hybrid information retrieval solution

Exploring the OpenAlex Data Structure and Visualization Author: Qingqin Fang (

 ORCID:

0009–0003–5348–4264)

 Introduction to OpenAlex

In today’s world, the realm of research papers is brimming with countless hot topics, and the sheer volume of publications can be overwhelming.

Unveiling Research Trends through OpenAlex Visualization

Neo4j APOC Library Use Case

 Author

Wenyi Pi (

 ORCID:

0009–0002–2884–277)

 Introduction

In the realm of Neo4j, the APOC (Awesome Procedures on Cypher) library stands as a powerful tool. Previously, We have talked about the importance of APOC in optimising Cypher queries and improving query efficiency in our article Exploring Methods of Cypher Query Optimisations.

Introduction to APOC: Enhancing Neo4j Capabilities

Unlocking the Future of AI: The Transformative Journey of Large Language Models Author · Vaibhav Khobragade (

 ORCID:

0009–0009–8807–5982)

 Introduction

Human language development is innate and evolves throughout life. Machines lack this ability to evolve without advanced AI algorithms.

The Journey of Large Language Models: Evolution, Application, and Limitations

Unlocking the Power of Questions — A deep dive into Question Answering Systems

Author:

 Amanda Kau (ORCID:


 0009–0004–4949–9284


 )

Virtual assistants have popped up on numerous websites over the years.

Harnessing The Power of Knowledge Graphs in Question Answering

Dhruv Gupta

Improving the performance of Large Language Models Author  Dhruv Gupta (ORCID: 0009-0004-7109-5403)  ChatGPT, which first came out in late 2022, took the world by storm. Since then, various LLM models and LLM based products such as Meta’s Llama and Google’s Gemini have emerged, demonstrating the power of LLMs.

RAG: The next big thing after LLMs?

Amir Aryani

Authors  Nakul Nambiar (ORCID: 0009-0009-9720-9233) Amir Aryani (ORCID: 0000-0002-4259-9774)  Knowledge graphs, which offer a structured representation of data and its relationships, are revolutionising how we organise and access information.

Mini RAG using Neo4j

Author  Amir Aryani (ORCID: 0000-0002-4259-9774) Introduction   In this article we look at Research Graph as an information model , and an approach to connect and capture the connections between research outputs, researchers and research activities. We explore the metadata model, and we discuss how to capture this graph in a Neo4j Graph Database.

Research Graph 101

Aland Astudillo

How to use GROBID to extract text from PDF  Author  Aland Astudillo (ORCID: 0009-0008-8672-3168)  GROBID is a powerful and useful tool based on machine learning that can extract text information from PDF files and other files to a structured format. One of the key challenges in knowledge mining from academic articles is reading the content of PDF files.

How to use GROBID

Unlocking the power of language models: A deep dive into BERT


 Author:


 Dhruv Gupta (ORCID:


 0009–0004–7109–5403


 )

Clive Humby, in 2006 rightly said, “Data is the new oil”. With data being present everywhere, it has never been more valuable.

Language Models: Deep Dive into BERT

An introduction to tools supporting PubMed Author: Xuzeng He (

 ORCID:

0009–0005–7317–7426) PubMed is a free Web literature search service developed and maintained by the National Center for Biotechnology Information (

 NCBI

), and it is also a part of NCBI’s Entrez retrieval system that provides access to a diverse set of 38 databases.

Enhancing PubMed: from a Medical Database to beyond

Exploring the potentials and limitations of Vision Language Models

Author:

 Amanda Kau (ORCID:


 0009–0004–4949–9284


 )

The human brain is more extraordinary than any machine we could build. From an early age, many of us gain the ability to comprehend what our eyes tell us and articulate it. Furthermore, we combine evidence from all our senses to reason.

How Much Can Vision Language Models Really “See”?

How to efficiently retrieve information for different applications

Author Wenyi Pi (ORCID: 0009-0002-2884-2771) This article aims to explore various ways in which Retrieval-Augmented Generation (RAG) can be utilised to retrieve information and generate responses effectively within the dialogue system. The rationale behind utilising RAG as well as potential ways in which it can be employed effectively will be covered.

Efficient Information Retrieval and Response Generation with Retrieval-Augmented Generation (RAG)

Addressing Misinformation, Bias, and Privacy Concerns

Author:

 Wenyi Pi (ORCID:


 0009–0002–2884–2771


 )

Recently, the spread of false or misinformation has become an important concern for people all over the world, especially in the era of big data and multimedia, where people are faced with a flood of information that makes it difficult to determine its authenticity.

The Ethical and Social Impact of Artificial Intelligence

Science in the age of large language models

Generative AI entails a credit–blame asymmetry

Exploring AI’s Ethical Terrain: Addressing Bias, Security, and Beyond

Author: Vaibhav Khobragade (

 ORCID:

0009–0009–8807–5982) Large language models (LLMs) like OpenAI’s GPT-4, Meta’s LLaMA, and Google Gemini (previously called Bard) have showcased their vast capabilities, from passing bar exams and crafting articles to generating images and website code.

Ethics and AI: Confronting the Challenges Ahead

The AI Helper Turning Mountains of Data into Bite-Sized Instructions

Author Aland Astudillo (ORCID: 0009-0008-8672-3168) LLMs have been changing the way the entire world deals with problems and day-by-day tasks. To make them better for specific applications, they need huge amounts of data and complex and expensive approaches to training them.

What is LLMLingua?

Zhuochen Wu

Authors:  Nakul Nambiar (ORCID: 0009-0009-9720-9233) Zhuochen Wu (ORCID: 0009-0000-5642-5348)  Research Graph is a structured representation of research objects that captures information about entities and the relationships between Researcher, Organisation, Publication, Grant and Research Data.

How to use GPT API to export a research graph from PDF publications

Tools and Platform for Integration of Knowledge Graph with RAG pipelines.

Authors Aland Astudillo (ORCID: 0009-0008-8672-3168) Aishwarya Nambissan (ORCID: 0009-0003-3823-6609) Many users of chatbots such as ChatGPT, have encountered the problem of receiving inappropriate or incompatible responses. There are several reasons why this might happen.

Unveiling the Synergy: Retrieval Augmented Generation (RAG) Meets Knowledge Graphs

Amir Aryani

A brief overview of different types of clustering techniques and their algorithms. Authors Aishwarya Nambissan (ORCID: 0009-0003-3823-6609) Amir Aryani (ORCID: 0000-0002-4259-9774)

 Background

Clustering is a fascinating technique used in machine learning, where patterns or data points are grouped based on their similarities. It’s like finding hidden connections among different data points without predefined labels.

19 Clustering Techniques

Unlocking the power of knowledge graphs in research catalogues: A deep dive into OpenAlex


 Author:


 Dhruv Gupta (ORCID:


 0009–0004–7109–5403


 )

Clive Humby, in 2006 rightly said, “Data is the new oil”. With data being present everywhere, it has never been more valuable.

Knowledge Graphs: Deep Dive into OpenAlex

Improving the performance and application of Large Language Models

Author Amanda Kau (ORCID: 0009-0004-4949-9284) Large language models (LLMs) like GPT-4, the engine of products like ChatGPT, have taken centre stage in recent years due to their astonishing capabilities. Yet, they are far from perfect.

Understanding Retrieval Pitfalls: Challenges Faced by Retrieval Augmented Generation (RAG) models

Yunzhong Zhang

Efficient creation of a stoplight report with data dashboard images

Author: Yunzhong Zhang (ORCID: 0009–0002–8177–419X) Comparing data dashboards is crucial for understanding trends and performance differences. Traditionally, this task required manual effort, which was slow and sometimes inaccurate. Now, thanks to OpenAI’s GPT-4 with Vision (GPT-4V), we are able to automate and improve this process.

How to use GPT-4V For Stoplight Report

Author Dhruv Gupta (

 ORCID

: 0009–0004–7109–5403) Introduction Large Language Models (LLMs) have become the new face of Natural language processing (NLP). With their generative power and ability to comprehend human language, the human reliance on these models is increasing every day. However, the LLMs have been known to hallucinate and thus produce wrong outputs.

Prompt engineering: A Way to Smartly Use AI

Recent Advances in Using Machine Learning with Graphs — Part 2 Latest findings in multiple research directions for handling graph construction and network security issues Author · Xuzeng He (

 ORCID:

0009–0005–7317–7426) Introduction A graph, in short, is a description of items linked by relations, where the items of a graph are called nodes (or vertices) and their relations are called edges (or links). Examples of

Recent Advances in Using Machine Learning with Graphs — Part 2

Bridging Human Perception and AI’s Future: The Convergence of Visual Understanding and Semantic Networks Author · Vaibhav Khobragade (

 ORCID:

0009–0009–8807–5982)

 Introduction

The fusion of Vision-Language Models (

 VLMs

), Generative Models, and Knowledge Graphs (

 KGs

) is reshaping how artificial intelligence (AI) understands and interacts with the world.

Tuning Vision-Language Models and Generative Models with Knowledge Graph

Using intelligence to use artificial Intelligence: A deep dive into Prompt Engineering


 Author


 Dhruv Gupta (ORCID:


 0009–0004–7109–5403


 )

Introduction Large Language Models (LLMs) have become the new normal in the field of Natural Language Processing (NLP). With their improved performance and generative power, people around the world are relying on it for

Prompt Engineering

An Overview of Constructing a Knowledge Graph Author · Qingqin Fang (

 ORCID:

0009–0003–5348–4264) 1. Introduction 1.1 What is a Knowledge Graph Knowledge Graphs are structured semantic knowledge bases used to rapidly describe concepts and their relationships in the physical world.

Unlocking Intelligence: The Journey from Data to Knowledge Graph

Boosting Performance for Knowledge Graphs with Neo4j APOC Library

 Author

Wenyi Pi (ORCID: 0009–0002–2884–2771) Introduction A

 knowledge graph

(graph database) captures information about main entities in a domain and the relationships between them. It was an augmented feature store for connected data which gave access to compute, access and operationalise structure features.

Exploring Methods of Cypher Query Optimisations

Aditya Iyengar

How AI is Transforming Medicine and Improving Lives
 
 AI in the Healthcare Domain
 


 Author

Aditya Iyengar (ORCID: 0009–0005–1959–9724)

 Introduction

The rapid integration of Artificial Intelligence (AI) into healthcare is no longer just a futuristic vision — it is happening now and at an extraordinary pace.

The Future of Healthcare

The Most Recent Cutting-Edge Models


 Author

Aditya Iyengar (ORCID: 0009–0005–1959–9724)

 Introduction

The past 6 months marked significant technological leaps in the field of artificial intelligence (AI), especially in the realm of image generation. In this six-month window, multiple innovations redefined the capabilities of text-to-image, video generation, and multimodal AI systems.

The New Era of AI Image Generation

Mingrui Gao

Working Examples


 Author

Mingrui Gao (

 ORCID

: 0009–0005–7271–2677)

 Introduction

Machine learning has revolutionised countless industries, but deploying ML models remains a significant challenge for many data scientists and developers.

What is Cog?

Let’s explore MLflow together. Author Wendi Fan (

 ORCID: 0000–0003–0284–9166

) Introduction MLflow is an open-source platform designed to manage the entire lifecycle of machine learning projects. It helps developers and data scientists streamline their workflows by tracking experiments, managing models, and deploying them efficiently.

MLflow Beginner’s Guide: How to Get Started with MLflow

Tarun Krishnan

Unleashing FLUX1.1 with the BFL API


 Author

Tarun Krishnan (

 ORCID

: 0009–0006–6647–127X)

 Introduction

With the growing demand for

 image generation models

in various applications, the need for

 APIs

that seamlessly integrate these models into developers’ workflows has skyrocketed.

The Fast Lane of Image Generation

Tohfa Siddika

The Godfather of AI Author Tohfa Siddika Barbhuiya (

 ORCID

: 0009–0007–2976–4601) In a monumental moment for the world of Artificial Intelligence (AI), Geoffrey Hinton, known as the “Godfather of AI,” has been awarded the Nobel Prize for his pioneering work in neural networks and deep learning. He has been awarded the 2024 Nobel Prize in physics by the Royal Swedish Academy of Sciences.

Geoffrey Hinton Wins Nobel Prize

Evaluating the Role of Self-Reflection in AI: Innovation or Overcomplication?

Author Tohfa Siddika Barbhuiya (

 ORCID

: 0009–0007–2976–4601) Introduction Artificial Intelligence (AI) has made remarkable strides over the past decade, with new advancements pushing the boundaries of what machines can achieve. One of the latest developments in the field is the introduction of “reflection” within AI models.

The Idea of Reflection Models

OpenAI Canvas and DevDay: Advancing AI Collaboration with Voice Realtime API, Model Distillation, and More Explore OpenAI’s latest tools and features — Canvas, Realtime API, Prompt Cache, Visual Fine-Tuning Author Zijian Yang (

 ORCID:

0009–0006–8301–7634) Introduction OpenAI recently introduced a new feature called Canvas, which is designed to provide ChatGPT users with a more powerful writing and coding collaboration

OpenAI Canvas and DevDay: Advancing AI Collaboration with Voice Realtime API, Model Distillation…

Working Examples Author Mingrui Gao (

 ORCID

: 0009–0005–7271–2677) Introduction In the rapidly evolving landscape of artificial intelligence, Large Language Models (LLMs) have emerged as powerful tools capable of understanding and generating human-like text. While cloud-based services offer convenient access to these models, there’s a growing demand for local, hands-on experimentation.

What is LM Studio?

A novel compression technique ensuring comparable performance with 70% less parameters

Author Amanda Kau (

 ORCID

: 0009–0004–4949–9284) Introduction The sizes of large language models (LLMs) have been steadily increasing over the last few years.

CompactifAI: Large Language Models Don’t Actually Have To Be Large

Let’s explore ChatGPT’s Advanced Voice Mode Author Wendi Fan (

 ORCID: 0000–0003–0284–9166

) Introduction In today’s fast-paced world, where convenience and efficiency are paramount, OpenAI’s latest addition to ChatGPT, the

 Advanced Voice Mode

, sets a new standard for human-computer interaction.

Unlocking the Power of ChatGPT’s Advanced Voice Mode

Automated Knowledge Graph Construction with Large Language Models — Part 2

 Harvesting the Power and Knowledge of Large Language Models

Author Amanda Kau (

 ORCID

:

 0009–0004–4949–9284

) Introduction Knowledge graphs (KGs) are a structured representation of data in a graphical format, in which entities are represented by nodes and are connected by edges representing relationships

Automated Knowledge Graph Construction with Large Language Models — Part 2

Understanding the Evolutionary Journey of LLMs Author Wenyi Pi (

 ORCID

: 0009–0002–2884–2771) Introduction When we talk about large language models (LLMs), we are actually referring to a type of advanced software that can communicate in a human-like manner. These models have the amazing ability to understand complex contexts and generate content that is coherent and has a human feel.

Brief Introduction to the History of Large Language Models (LLMs)

An improvement architecture superior to the Transformer, proposed by Meta Author · Qingqin Fang (

 ORCID:

0009–0003–5348–4264) Introduction Recently, researchers from Meta and the University of Southern California have introduced a model called Megalodon. They claim that this model can expand the context window of language models to handle millions of tokens without overwhelming your memory.

The longer the context, the better? Unlimited Context Length in Megalodon

Attention mechanism not getting enough attention Author Dhruv Gupta (

 ORCID

: 0009–0004–7109–5403) Introduction As discussed in this article, RNNs were incapable of learning long-term dependencies. To solve this issue both LSTMs and GRUs were introduced. However, even though LSTMs and GRUs did a fairly decent job for textual data they did not perform well.

Transformers Models in NLP

The Three Oldest Pillars of NLP

Author Dhruv Gupta (

 ORCID

: 0009–0004–7109–5403) Introduction Natural Language Processing (NLP) has almost become synonymous with Large Language Models (LLMs), Generative AI, and fancy chatbots. With the ever-increasing amount of textual data and exponential growth in computational knowledge, these models are improving every day.

RNNs vs GRUs vs LSTMs

Large Language Models for Fake News Generation and Detection Author Amanda Kau (

 ORCID

: 0009–0004–4949–9284) Introduction In recent years, fake news has become an increasing concern for many, and for good reason. Newspapers, which we once trusted to deliver credible news through accountable journalists, are vanishing en masse along with their writers.

Are Large Language Models Our Allies or Enemies in the Fight Against Fake News?

A Unified and Collaborative Framework for LLM Author · Qingqin Fang (

 ORCID:

0009–0003–5348–4264) Introduction In today’s rapidly evolving field of artificial intelligence, large language models (LLMs) are demonstrating unprecedented potential. Particularly, the Retrieval-Augmented Generation (RAG) architecture has become a hot topic in AI technology due to its unique technical capabilities.

RAG 2.0 is Coming?

Exploring the Potential of Temporal Feature-Logic Embedding (TFLEX) in Complex Query Resolution

Author · Vaibhav Khobragade (

 ORCID:

0009–0009–8807–5982)

 Introduction

Artificial intelligence (AI) and knowledge representation in the field of temporal knowledge graphs are rapidly gaining interest.

Harnessing Temporal Dynamics: Advanced Reasoning using Temporal Knowledge Graphs

Integrating temporal data into static knowledge graphs

 Author

Amanda Kau

 (ORCID:


 0009–0004–4949–9284


 )

Introduction Knowledge graphs (KGs) have proven to be an effective method of data representation that is increasingly popular. In KGs, entities and concepts are represented as nodes, while the relationships between nodes are depicted as edges.

Dynamic Knowledge Graphs: A Next Step For Data Representation?

Understanding the Balance between Internal Knowledge and External Sources

Author Qingqin Fang (

 ORCID:

0009–0003–5348–4264) Introduction Previous research often emphasized the limitations of LLM’s information acquisition pathways, focusing on enhancing its capabilities in this regard.

When RAG and LLM Conflict: Who Will AI Listen to?

Exploring the Boundaries of Creativity and Responsibility in the Age of AI-Driven Media

 Author

Vaibhav Khobragade (

 ORCID:

0009–0009–8807–5982)

 Introduction

In 2024, the discipline of Generative AI takes a big step forward with the launch of revolutionary models that convert text into dynamic films, altering the landscape of digital content creation.

Generative AI’s Leap into Video in 2024 and its Ethical Horizon

Author  Amir Aryani: (ORCID: 0000-0002-4259-9774) Definition   A research collaboration network is a group of researchers, and practitioners, or both, working together on joint research activities. These networks often span across disciplines, geographic boundaries, and sectors, enabling participants to share resources, expertise, and data to address common research goals more effectively than they could individually.

Research Collaboration Network

Understanding Sequential Data Modelling with Keras for Time Series Prediction

 Author

Wenyi Pi (

 ORCID

: 0009–0002–2884–2771) Introduction Recurrent Neural Networks (RNNs) are a special type of neural networks that are suitable for learning representations of sequential data like text in Natural Language Processing (NLP). We will walk through a complete example of using RNNs for time series prediction, covering

Beginner’s Guide to Recurrent Neural Networks (RNNs) with Keras

Incorporating Knowledge Graphs to explain reasoning processes


 Author

Amanda Kau (

 ORCID:

0009–0004–4949–9284) Introduction Large language models (LLMs) like GPT-4 possess remarkable language abilities, allowing them to function as chatbots, translators, and much more.

Combining Knowledge Graphs With Language Models for Interpretability

Latest findings in multiple research directions for handling graph prediction and optimization Author · Xuzeng He (

 ORCID:

0009–0005–7317–7426) A graph, in short, is a description of items linked by relations, where the items of a graph are called

 nodes

(or vertices) and their relations are called

 edges

(or links). Examples of graphs can include social networks (e.g. Instagram) or knowledge

Recent Advances in using Machine Learning with Graphs

Prompt Engineering — Part 2


 Using intelligence to use artificial Intelligence: A deep dive into Prompt Engineering

Author Dhruv Gupta

 (ORCID:

0009–0004–7109–5403

 )

Introduction In the previous article we discussed what prompt engineering and some of the techniques used for prompt engineering.

Prompt Engineering — Part 2

Using Mistral for Data tagging Author · Xuzeng He (

 ORCID:

0009–0005–7317–7426) Introduction Data tagging, in simple terms, is the process of assigning labels or tags to your data so that they are easier to retrieve or analyse.

How to use Large Language Models to tag your data: A complete tutorial

Ministral — The World’s Best Edge Language Model
 


 Revolutionising On-Device AI: High-Performance, Low-Resource Language Models for Edge Computing


 Author

Tarun Krishnan (

 ORCID

: 0009–0006–6647–127X)

 Introduction

In recent years, there’s been a notable shift in the AI landscape, with many companies moving away from developing massive,

Ministral — The World’s Best Edge Language Model

Author: Vaibhav Khobragade (

 ORCID:

0009–0009–8807–5982) Introduction Large language models (LLMs) are becoming increasingly popular in natural language processing for their superior competence in various applications.

Enhancing Language Models: The Role of Knowledge Graph Augmentation in Overcoming LLM Challenges

An Introduction to RA-CM3, MuRAG and RACE Author  Xuzeng He (ORCID: 0009-0005-7317-7426)  Generative Artificial Intelligence (GAI) has demonstrated impressive performances in tasks such as text generation and text-to-image generation.

Recent Advances in using Retrieving Multimodal Information for Augmented Generation

Understanding Knowledge Networks From A Graph Perspective


 Author:


 Amanda Kau (ORCID:


 0009–0004–4949–9284


 )

Since 2020, over ten million scholarly articles have been published annually. To put that into perspective, say all ten million articles were released on the first day of the year.

Uncovering The Secrets Behind The Most Influential Scholarly Publications With AceMap

Latest effort in assessing the security of the code generated by large language models Author · Xuzeng He (

 ORCID:

0009–0005–7317–7426) Introduction With the surge of Large Language Models (LLMs) nowadays, there is a rising trend among developers to use Large Language Models to assist their daily code writing. Famous products include GitHub Copilot or simply ChatGPT.

Large Language Models for Code Writing: Security Assessment

Harvesting the Power and Knowledge of Large Language Models


 Author

Amanda Kau

 (ORCID:

0009–0004–4949–9284

 )

Introduction Knowledge Graphs are networks that represent data in a graphical format. The beauty of Knowledge Graphs lies in their representation of concepts, events and entities as nodes, and the relationships between them as edges.

Automated Knowledge Graph Construction with Large Language Models

Understanding how RNNs work and its applications


 Author

Wenyi Pi (

 ORCID

: 0009–0002–2884–2771) Introduction In the ever-evolving landscape of artificial intelligence (AI), bridging the gap between humans and machines has seen remarkable progress. Researchers and enthusiasts alike have tirelessly worked across numerous aspects of this field, bringing about amazing advancements.

An Introduction to Recurrent Neural Networks (RNNs)

Latest findings for the use of Knowledge Graph in the field of QA in multiple research directions Author: Xuzeng He (

 ORCID:

0009–0005–7317–7426) Question Answering (QA), the ability to interact with data using natural language questions and obtaining accurate results, has been a long-standing challenge in computer science dating back to the 1960s.

Question Answering enhanced by Knowledge Graph

Latest findings in pre-training graphs and using them for link recommendation Author · Xuzeng He (

 ORCID:

0009–0005–7317–7426) Introduction A graph, in short, is a description of items linked by relations, where the items of a graph are called nodes (or vertices) and their relations are called edges (or links). Examples of graphs can include social networks (e.g. Instagram) or knowledge graphs (e.g. Wikipedia). In Instagram

Working with Graphs: Pre-training and Application

Anthropic’s Groundbreaking AI, Claude 3.5, Uses Computers Like a Human and Becomes a Game-Changer in Automation


 Author

Zijian Yang (

 ORCID:

0009–0006–8301–7634)

 Introduction

Claude 3.5 Receives a Major Upgrade Overnight! As anticipated, Anthropic AI made a significant move this week with the launch of Claude 3.5 Haiku.

New Claude 3.5 Can Control Computer: Outsmarts o1 in Coding and Redefines Agent Capabilities

Solutions to Enhance LLM Performance in Long Contexts Author · Qingqin Fang (

 ORCID:

0009–0003–5348–4264) Introduction In the era of AI breakthroughs, large language models (LLMs) are not just advancements; they are revolutions, transforming how we interact with technology, from casual conversations with chatbots to the intricate mechanisms behind sophisticated data analysis tools.

Navigating the Long Context Conundrum: Challenges in Language Models’ Information Processing

Supervised Fine-tuning, Reinforcement Learning from Human Feedback and the latest SteerLM Author · Xuzeng He (

 ORCID:

0009–0005–7317–7426) Introduction Large Language Models (LLMs), usually trained with extensive text data, can demonstrate remarkable capabilities in handling various tasks with state-of-the-art performance. However, people nowadays typically want something more personalised instead of a general solution.

Fine-tuning Large Language Models: A Brief Introduction

How Apple’s Research Exposes AI’s Inability to Reason and OpenAI Highlights ChatGPT’s Name-Based Biases


 Author

Zijian Yang (

 ORCID:

0009–0006–8301–7634)

 Introduction

As artificial intelligence continues to evolve, the capabilities and limitations of large language models (LLMs) are under increasing scrutiny.

ChatGPT’s Name Bias and Apple’s Findings on AI’s Lack of Reasoning: Major Flaws Revealed

Working Examples


 Author

Mingrui Gao (

 ORCID

: 0009–0005–7271–2677)

 Introduction

In the fast-paced world of artificial intelligence, a new platform is gaining attention by making AI-powered content creation more accessible.

What is Glif?

Stability AI’s latest release offering New Models and Features


 Author

Tohfa Siddika Barbhuiya (

 ORCID

: 0009–0007–2976–4601)

 Stable Diffusion 3.5

by

 Stability AI

marks a major step forward in AI-driven image generation.

Stable Diffusion 3.5

Research Graph

Google’s Groundbreaking AI Model Explained

Introduction In the rapidly evolving landscape of artificial intelligence, Google continues to stand at the forefront, consistently pushing the boundaries of what these technologies can achieve. One of their latest advancements, GEMMA2, represents a significant leap in AI capabilities. Let’s delve into the details of this new AI model by Google.

What is Gemma2?

Enhancing Open-Domain Conversational Question Answering with Knowledge-Enhanced Models and Knowledge Graphs


 How knowledge-enhanced language models and knowledge graphs are advancing open-domain conversational question answering

Author:

 Wenyi Pi (ORCID:


 0009-0002-2884-2771


 )

When searching for information on the website, it is common to come across a flood of

Enhancing Open-Domain Conversational Question Answering with Knowledge-Enhanced Models and…

Refining AI Vision: How Retrieval-Augmented Generation Transforms Image Captioning in Large Language Models   Leveraging External Knowledge to Enhance the Descriptive Capabilities of AI Systems Author  Vaibhav Khobragade (ORCID: 0009–0009–8807–5982) Introduction   Large Language Models (LLMs) are artificial intelligence models that are trained on massive amounts of text data in order to generate human-like language and produce coherent

Refining AI Vision: How Retrieval-Augmented Generation Transforms Image Captioning in Large…

An Introduction to Retrieval Augmented Generation (RAG) and Knowledge Graph

Author Qingqin Fang (ORCID: 0009–0003–5348–4264)

 Introduction

Large Language Models (LLMs) have transformed the landscape of natural language processing, demonstrating exceptional proficiency in generating text that closely resembles human language.

Hallucination in Large Language Models and Two Effective Alleviation Pathways

Stories by Research Graph on Medium

### Le Chat by Mistral: Access ChatGPT-Like Features, Image Generation, Canvas, Artifact, and More for Free

#### Discover Le Chat: Free ChatGPT Alternatives with Canvas, Internet Search, and Flux.1 Image Generation {#discover-le-chat-free-chatgpt-alternatives-with-canvas-internet-search-and-flux.1-image-generation}

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*kRNopWaOIAN_WDw-.png" />
</figure>

### Author

- [Zijian Yang](https://www.linkedin.com/in/zijian-yang/) (**ORCID:**
 [0009--0006--8301--7634](https://orcid.org/0009-0006-8301-7634))

### Introduction

Imagine accessing premium ChatGPT features --- Canvas, internet search,
and PDF analysis --- completely free of charge. Thanks to Mistral AI's
latest updates to their platform, [**Le
Chat**](https://chat.mistral.ai/chat), this is now a reality. Not only
does Le Chat offer cutting-edge functionality that rivals ChatGPT, but
it also integrates advanced image generation capabilities through a
collaboration with Flux, bringing all these tools into a single,
cost-free platform.

In this article, we dive into how Mistral's Le Chat stacks up against
competitors, explore its breakthrough multimodal model Pixtral Large,
and uncover its strengths and limitations. Whether you're curious about
leveraging these tools for business or research or just exploring
alternatives to paid AI platforms, this is your definitive guide to the
future of AI accessibility. Try it now at
[**chat.mistral.ai/chat**](https://chat.mistral.ai/chat).

### Main Content

Mistral AI has introduced new features to its Le Chat platform,
including Canvas, internet search, and PDF uploads, all mirroring
ChatGPT's capabilities --- and they are **completely free!**

Now, you can access the features of three major AI models all in
one place.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*t4LD4RooNm-Gg9jb.png" />
</figure>

How le Chat compares to leading chat and search assistants in the
market today

They also announced a partnership with the renowned image generation
model Flux, offering online image creation capabilities on the Le Chat
platform, again at no cost.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/1*sTDFfovD4nUw5fawJgxJHQ.png" />
<figcaption><a
href="https://x.com/bfl_ml/status/1858572519571063076">Source</a></figcaption>
</figure>

Mistral's CEO, Arthur Mensch, stated that the new version of Le Chat is
a milestone, marking the platform as a "more mature animal."

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/1*AyT5g4BgGf2YfshziOmJ-Q.png" />
<figcaption><a
href="https://x.com/arthurmensch/status/1858567024609276372">Source</a></figcaption>
</figure>

### ChatGPT-Like Features Available for Free

On Mistral's official platform, Le Chat (which may sound like "chat" but
actually means "cat" in French), a new series of ChatGPT-like features
have been launched for free.

According to a table released by Mistral, aside from o1 and voice
conversations, most other features are included.

This also encompasses OpenAI's newly launched advanced interface,
Canvas, which is currently exclusive to ChatGPT subscribers.

- Access to the latest frontier model
- Web search with citations
- Canvas for ideation, in-line editing, and export
- Image generation
- Image/document analysis
- Task agents
- Single platform from models to outputs

In Le Chat's Canvas, generated documents can be displayed separately,
eliminating the need to manually extract them from conversations.

<figure>
<img
src="https://cdn-images-1.medium.com/max/772/0*rPa8Cl58J9Mp7Vc1.gif" />
</figure>

Code can also be handled in the same way, with support for real-time
previews and the ability to select specific sections for modification.

<figure>
<img
src="https://cdn-images-1.medium.com/max/772/0*xSwrKv7nNeH_x8Nh.gif" />
</figure>

Next is the internet search capability. Like ChatGPT, Le Chat can
perform online searches to summarise information and provide source
citations, all at a very fast speed.

<figure>
<img
src="https://cdn-images-1.medium.com/max/738/0*SOMNXlw6lql-Hp_h.gif" />
</figure>

Additionally, the current version of Le Chat can process complex PDF
documents and images, performing analysis and summarisation of tables,
charts, text, formulas, equations, and more.

In the following example, Le Chat extracted information, summarised, and
semantically understood the famous quantum entanglement paper written by
Einstein and others in 1935.

<figure>
<img
src="https://cdn-images-1.medium.com/max/766/0*VgqSpS0cNuHK0D9q.gif" />
</figure>

In addition to image analysis, the latest version of Le Chat also
supports text-to-image generation. However, Mistral has not developed
these models independently and has instead partnered with Flux to
integrate their models.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*19eH9fZkwqBCnxHS.png" />
</figure>

It is also possible to create and @Agent in conversations to
intelligently perform certain tasks, but currently, the creation page is
only available in French.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*mUOrenAr9R_2hGNN.png" />
</figure>

However, the models released by Mistral so far have not extended to
voice modalities, so Le Chat does not have the advanced voice
interaction features available in ChatGPT. Additionally, there is no
desktop client.

Therefore, although Mistral's efforts are seen as an attempt to catch up
with ChatGPT in terms of features, there is still some distance to go
before they truly match it. However, Mistral does have the advantage of
being free.

### Open-Source Multimodal Model: Pixtral Large

Pixtral Large can be used for research and educational purposes under
the Mistral Research License (MRL), and for experimental, testing, and
production purposes in commercial settings under the Mistral Commercial
License.

Pixtral Large is an evolution of Mistral Large 2, released in the summer
of 2024, and the first multimodal model Pixtral 12-B, released in
September. Regarding the expansion of the multimodal model to 124
billion parameters, Mistral AI's CEO stated: "We increasingly recognize
that creating the best AI experience requires co-designing the model and
the product interface. Pixtral was trained with high-impact front-end
applications in mind, making it a great example."

Pixtral Large includes a 123 billion parameter decoder and a 1 billion
parameter visual encoder, excelling in both text and visual data
processing.

With a context window of 128K, Pixtral Large can handle at least 30
high-resolution images or approximately a 300-page book, comparable to
the capabilities of leading models in the OpenAI GPT series.

In terms of performance, the model demonstrates state-of-the-art results
on various benchmarks, including MathVista, DocVQA, and VQAv2, making it
highly suitable for tasks such as chart interpretation, document
analysis, and image understanding.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/1*-ci132q5MXsWWxo3aSq89A.png" />
<figcaption>Comparison of performance on various benchmarks</figcaption>
</figure>

Specifically, on the MathVista benchmark, Pixtral Large achieved an
accuracy of 69.4%, outperforming all other models. In the ChartQA and
DocVQA benchmarks, Pixtral Large surpassed GPT-4o and Gemini-1.5 Pro.

Pixtral Large also demonstrated strong competitiveness on the
MM-MT-Bench, outperforming Claude-3.5 Sonnet (the new version),
Gemini-1.5 Pro, and GPT-4o (the latest version).

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/1*pERFknnD5oKqs5oGRflYtA.png" />
<figcaption>Model evaluations</figcaption>
</figure>

In terms of image understanding, Pixtral Large also performs
exceptionally well. For instance, when you upload a bill and ask the
model, "I bought coffee and sausage, plus an 18% tip. How much should I
pay?" it can accurately calculate the total amount.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*9r-poDHYyrGgN2A4.png" />
</figure>

Pixtral Large will provide the total amount in an organised manner,
first calculating the cost of the coffee and sausage --- such as two
lattes and one sausage --- then computing the 18% tip, and finally
providing the total amount.

<figure>
<img
src="https://cdn-images-1.medium.com/max/742/0*DihYuinTNjfC-hTC.png" />
</figure>

Pixtral Large can also accurately understand and analyse charts. For
instance, when asked about the point at which the "dark dragon" model
starts experiencing issues based on a training loss curve, Pixtral Large
accurately analyses: "At 10,000 steps, the training loss begins to show
instability... Subsequently, this instability persists, with another
significant peak occurring around 20,000 steps..."

In the next example, when you upload an image and ask Pixtral Large
which companies use Mistral AI models, it can provide an accurate
response based on the information presented in the image.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/1*Axytihp-QoY7DVWItXqIew.png" />
</figure>

Pixtral Large can provide accurate results based on the information in
the image, and its response process is as follows.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/1*TP2KeIXsWS03eWHIwzuYyQ.png" />
</figure>

However, on Reddit, some have questioned whether Pixtral truly achieves
state-of-the-art performance. They point out that Mistral's official
comparisons were limited to a few models, **including only the
open-source Llama-3.2 90B**.

In reality, on multiple datasets, Qwen2-VL (with a maximum parameter
size of 72B) performs better than Pixtral. Additionally, on certain
datasets, Pixtral's test scores are also lower than those of Molmo,
developed by a non-profit research organisation called Ai2 based
in Seattle.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*C_pXB653Dfht-_1x.png" />
<figcaption><a
href="https://www.reddit.com/r/LocalLLaMA/comments/1gu7cm8/comment/lxs7gbe/?utm_source=share&amp;utm_medium=web3x&amp;utm_name=web3xcss&amp;utm_term=1&amp;utm_content=share_button">Source</a></figcaption>
</figure>

Moreover, some users have reported that in their tests involving images
with Japanese text, Pixtral Large's recognition capabilities were
inferior to the 7B version of Qwen.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*NXylhUwOFzsUGDXb.png" />
<figcaption><a
href="https://x.com/AiXsatoshi/status/1858648441616035951">Source</a></figcaption>
</figure>

In addition to Pixtral Large, Mistral AI's most advanced text model,
Mistral Large, has also received an update. This model is available via
the API as pixtral-large-latest and on Hugging Face as Mistral
Large 24.11.

Mistral Large 24.11 will first be rolled out on Google Cloud and
Microsoft Azure, with availability expected within a week.

- Model and weights download link:
 <https://huggingface.co/mistralai/Pixtral-Large-Instruct-2411/tree/main>
- Trial link: <https://chat.mistral.ai/chat>

### Conclusion

Mistral AI is reshaping the AI landscape with its accessible and
innovative platform, **Le Chat**. By offering premium features like
Canvas, internet search, and PDF analysis for free, and integrating
advanced capabilities like image generation through partnerships,
Mistral is positioning itself as a strong competitor in the AI assistant
market. The launch of its multimodal model, **Pixtral Large**, further
highlights its commitment to pushing the boundaries of AI performance.

While there are still areas where Mistral lags behind competitors like
ChatGPT --- such as voice interactions and desktop clients --- the
platform's affordability and feature set make it a compelling option for
both casual and professional users. Whether you're exploring AI for
research, productivity, or creative endeavours, Mistral's Le Chat and
Pixtral Large offer a glimpse into a future where advanced AI tools are
accessible to all.

Explore Mistral's innovations firsthand at
[**chat.mistral.ai/chat**](https://chat.mistral.ai/chat) and experience
the growing potential of open AI ecosystems.

### References

- AI, Mistral. "Pixtral Large." *Mistral.ai*, 18 Nov. 2024,
 mistral.ai/news/pixtral-large/. Accessed 20 Nov. 2024.
- AI, Mistral. "Mistral Has Entered the Chat." *Mistral.ai*, 18 Nov.
 2024, mistral.ai/news/mistral-chat/. Accessed 20 Nov. 2024.
- "Reddit --- Dive into Anything." *Reddit.com*, 2024,
 [www.reddit.com/r/LocalLLaMA/comments/1gu7cm8/mistral_large_2411_and_pixtral_large_reLease_18th/.](http://www.reddit.com/r/LocalLLaMA/comments/1gu7cm8/mistral_large_2411_and_pixtral_large_reLease_18th/.)
 Accessed 20 Nov. 2024.

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3491ec9901ec){width="1"
height="1"}

### OpenAI Canvas and DevDay: Advancing AI Collaboration with Voice Realtime API, Model Distillation, and More

#### Explore OpenAI's latest tools and features --- Canvas, Realtime API, Prompt Cache, Visual Fine-Tuning {#explore-openais-latest-tools-and-features-canvas-realtime-api-prompt-cache-visual-fine-tuning}

<figure>
<img
src="https://cdn-images-1.medium.com/max/800/0*EdCGEumO7YJU66dB.png" />
</figure>

### Author

- [Zijian Yang](https://www.linkedin.com/in/zijian-yang/) (**ORCID:**
 [0009--0006--8301--7634](https://orcid.org/0009-0006-8301-7634))

### Introduction

OpenAI recently introduced a new feature called Canvas, which is
designed to provide ChatGPT users with a more powerful writing and
coding collaboration experience. Canvas is a newly crafted interface
that offers an independent workspace beyond the usual ChatGPT dialogue,
built on the latest GPT-4o model to support more complex writing and
coding projects.

Canvas focuses on two main categories: writing and coding. On the
writing side, Canvas offers editing suggestions, adjustable text length,
readability level options, and the ability to add final touches,
including emojis. On the coding side, Canvas provides code review,
annotation and logging, bug fixing, and the capability to convert code
into other programming languages such as JavaScript or Python. These
features allow users to co-create and refine ideas with AI, making
collaboration more seamless and effective.

In essence, Canvas introduces a novel way for users and AI to work side
by side --- not just through simple dialogue, but as genuine
collaborators. This helps users precisely address and optimise content
within the project's context, marking yet another step by OpenAI towards
enhancing productivity with AI, aiming for a more integrated and smooth
collaborative experience.

At the recent DevDay, OpenAI also announced five major innovations:
Realtime API, Prompt Cache, Model Distillation, Visual Fine-Tuning, and
New Playground Features. These updates signal a shift in OpenAI's
strategic focus towards infrastructure and the developer ecosystem,
rather than solely competing in the end-user AI application market.

Compared to last year's grand reveal, this year's DevDay was more
understated, focusing on incremental improvements to existing AI tools
and API suites while showcasing enhanced developer capabilities and
community stories. Here's a brief overview of the five updates:

- **Realtime API**: Offers developers a near real-time
 "speech-to-speech" experience, with six voice options provided
 by OpenAI.
- **Prompt Cache**: Similar to the feature introduced by Anthropic
 earlier, this allows developers to cache common contexts between API
 calls, reducing costs and improving latency.
- **Model Distillation**: Developers can fine-tune smaller models (such
 as GPT-4o mini) using larger models like o1-preview and GPT-4o.
- **Visual Fine-Tuning**: Combines images and text to fine-tune GPT-4o
 applications, significantly enhancing the model's visual
 understanding.
- **New Playground Features**: Offers a new framework and usage options
 for prompts, along with breakthroughs in structured output, making
 development easier than ever.

### Canvas: Collaborate Side-by-Side with ChatGPT

Canvas is a new feature of ChatGPT that provides an independent
workspace, allowing users to interact with ChatGPT in a more intuitive
and flexible way. Built on OpenAI's latest GPT-4o model, Canvas
overcomes the limitations of traditional chat windows for handling
complex tasks. Whether it's writing or coding, this new collaborative
approach offers an enhanced experience.

Currently, Canvas is in its testing phase and can be manually selected
across all models. Plus users have immediate access without any wait,
while OpenAI plans to eventually roll it out to all free users. With
Canvas, you can do research, write code, draft emails, and most
importantly, brainstorm creatively alongside ChatGPT.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*MUxNob5kVs7SruI6.png" />
</figure>

In Canvas mode, it's no longer just about having a simple conversation.
Instead, users can refine, rework, and organise the generated
content --- essentially transforming the process into something akin to
a content or code editor.

> For instance, if you upload a file and ChatGPT generates some content,
> you'll notice an **"Edit"** button in the **bottom right corner** of
> the interface, allowing you to take full control of the generated
> output.

<figure>
<img
src="https://cdn-images-1.medium.com/max/796/0*hcqH_5mn7LInMrTw.gif" />
</figure>

This button reveals **five features**, which include:

- **Suggest edits**
- **Adjust the length**
- **Reading level**
- **Add final polish**
- **Add emojis**

For example, with the "Suggest edits" feature, you can simply select the
portion of text you want to modify, then click **"Apply"** to have the
content automatically regenerated.

<figure>
<img
src="https://cdn-images-1.medium.com/max/796/0*St5DNZeIrNy0_96c.gif" />
</figure>

You can also "highlight" the article title and have it "reworked" to
your requirements, with fine-tuning options available.

<figure>
<img
src="https://cdn-images-1.medium.com/max/796/0*Hxel9Mr9Kzq9Vg8S.gif" />
</figure>

You can also highlight specific sections to draw ChatGPT's attention,
allowing the model to provide in-line feedback and suggestions while
considering the overall project context.

<figure>
<img
src="https://cdn-images-1.medium.com/max/783/0*gfvVpJ2juJQjkj8T.gif" />
</figure>

So, what else can you do with Canvas? Let's continue exploring.

### Writing with ChatGPT

Earlier, we showcased the "Edit" feature within Canvas writing. Now,
let's take a look at the **"Adjust the Length"** option.

As the name suggests, this feature allows you to adjust the length of
the document, making it either shorter or longer. There are five
settings available: Current Length, Longer, Longest, Shorter, and
Shortest.

<figure>
<img
src="https://cdn-images-1.medium.com/max/798/0*IIDxo9DLmQ0kV4e2.gif" />
</figure>

Once you select the desired length, ChatGPT will adjust the entire text,
modifying it word by word and paragraph by paragraph.

<figure>
<img
src="https://cdn-images-1.medium.com/max/797/0*NIA4KtoIfzhNZSg5.gif" />
</figure>

The **"Change Reading Level"** feature is quite impressive, offering six
levels: **Kindergarten**, **Middle School**, **Current Level**, **High
School**, **College**, and **Graduate**.

Simply select the desired reading level, and ChatGPT will adjust the
entire text accordingly.

<figure>
<img
src="https://cdn-images-1.medium.com/max/796/0*WUqaStUDESwRCVmo.gif" />
</figure>

The **"Add Final Polish"** feature makes comprehensive edits to the
entire text, including checking grammar, clarity, and consistency.

<figure>
<img
src="https://cdn-images-1.medium.com/max/640/0*0W4aleahXJVCiUiX.gif" />
</figure>

The **"Add Emojis"** feature is quite fun --- it lets you add emojis to
your text, making the content more lively and engaging.

For example, one user shared their creative result:

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*qYPgOMC6QCMiSugc.png" />
</figure>

### Coding with ChatGPT

In addition to writing, Canvas also offers a **coding** feature, which
includes the following five capabilities:

- **Review code**
- **Add logs**
- **Add comments**
- **Fix bugs**
- **Port to a language**

<figure>
<img
src="https://cdn-images-1.medium.com/max/640/0*a6pDeE2I_zpiU_oB.gif" />
</figure>

For example, let's give ChatGPT a request:

> *Help me write an API web server in Rust.*

<figure>
<img
src="https://cdn-images-1.medium.com/max/785/0*UaPIZKjAXIIibE7b.gif" />
</figure>

In Canvas, you can highlight a specific code snippet and request
modifications according to your requirements.

<figure>
<img
src="https://cdn-images-1.medium.com/max/785/0*ncuffX5z3PCfLHFy.gif" />
</figure>

For specific details, you also have the option to manually make changes.

<figure>
<img
src="https://cdn-images-1.medium.com/max/785/0*DvGxuBB406PHyaxk.gif" />
</figure>

For features like code review, adding logs, adding comments, and fixing
bugs, a single click allows you to make global changes.

Notably, the **Change Programming Language** feature currently offers
options including JavaScript, TypeScript, Python, Java, C++, and PHP.

<figure>
<img
src="https://cdn-images-1.medium.com/max/802/0*LRUC3mXVyWGUUXZm.gif" />
</figure>

### Making AI the Ultimate Collaborator

For Canvas, OpenAI has implemented over 20 automated internal
evaluations to measure progress, utilising novel synthetic data
generation techniques --- such as extracting outputs from OpenAI
o1-preview --- to conduct post-training on the model for its core
behaviours.

The advantage of this approach is the ability to quickly address writing
quality and new user interaction issues without relying on manually
generated data.

A key challenge is defining when to trigger Canvas. The model needs to
be sensitive enough to identify tasks requiring detailed review or
modification, like "write a blog post about the history of coffee
beans," while avoiding unnecessary activations for more general tasks,
such as "help me create a new dinner recipe," which might not
require Canvas.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*bnJNDOycfTYpq5wo.png" />
</figure>

For writing and coding tasks, we improved correctly triggering the
canvas decision boundary, reaching 83% and 94% respectively compared to
a baseline zero-shot GPT-4o with prompted instructions.

The second challenge involves adjusting the model's editing behaviour
once Canvas is triggered, particularly deciding when to make targeted
edits instead of rewriting the entire content.

The model is trained to favour targeted edits, especially when the user
explicitly selects text. As the model continues to evolve, its ability
to handle these nuanced editing tasks is also improving.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*nEUWtqX3dkqXqF5w.png" />
</figure>

For writing and coding tasks, we prioritised improving canvas-targeted
edits. GPT-4o with canvas performs better than a baseline prompted
GPT-4o by 18%.

Lastly, training the model to generate high-quality code comments also
requires a detailed iterative process. This evaluation differs from the
previous cases. While determining whether Canvas was triggered or
whether targeted edits were made can be easily automated, assessing the
quality of code comments is more challenging to automate, leading the
team to opt for manual evaluation.

Compared to the baseline model, the integration of Canvas improved
GPT-4o's comment accuracy by 30% and overall quality by 16%. This
indicates that training with synthetic data significantly enhances the
model's response quality and behaviour, compared to zero-shot prompts
with detailed instructions.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*2gLX1QEl6PUDaTug.png" />
</figure>

Human evaluations assessed canvas comment quality and accuracy
functionality. Our canvas model outperforms the zero-shot GPT-4o with
prompted instructions by 30% in accuracy and 16% in quality.

A developer described the Canvas interface as a game changer. They
recently used Canvas with ThreeJS to create a tesseract/Hypercube
visualisation tool and mentioned how much they enjoy the unified UX for
chatting, live reviews and watching GPT-4o work its magic on
code --- all in one place, always relevant.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*B8xVKylexLVk6BBV.png" />
</figure>

<figure>
<img
src="https://cdn-images-1.medium.com/max/960/0*9sqqQVDeyDnGc9Vz.gif" />
</figure>

### Realtime Multimodal Conversation API: The Era of AI Voice Applications Has Arrived!

OpenAI's newly launched Realtime API is **currently in public beta**.

This new product allows developers to create low-latency, multimodal
experiences, particularly for voice-to-voice applications.

This means developers can now start adding ChatGPT's "voice controls" to
their apps.

To showcase the API's potential, OpenAI demonstrated an updated version
of Wanderlust --- a "travel planning" app originally presented at last
year's event. With the Realtime API, users can interact with the app
directly, planning trips in a natural, conversational manner. The system
even supports interruptions, mimicking the pauses and nuances of human
conversation.

While the travel planning app is just an example, the Realtime API opens
up a wide range of possibilities for voice applications across various
industries.

From **customer service** to **education** and **assistive tools**,
developers now have a powerful new resource for creating more intuitive,
responsive AI-driven experiences.

The Realtime API essentially simplifies the process of building **voice
assistants** and other **conversational AI tools**, eliminating the need
to piece together multiple models for transcription, inference, and
text-to-speech conversion.

Early adopters of the Realtime API, such as Healthify --- a nutrition
and fitness guidance app --- and Speak, a language learning platform,
have already integrated the API into their products. These
implementations demonstrate the potential of the API to create more
natural, engaging user experiences in fields like healthcare and
education.

Though the Realtime API **is not cheap (\$0.06 per minute for audio
input, \$0.24 per minute for audio output)**, it still represents a
significant value proposition for those looking to develop voice-based
applications.

### Prompt Cache: A Major Budget Saver for Developers

The "Prompt Cache" feature is **designed to reduce costs and latency for
developers**.

The system automatically applies a 50% discount to tokens that the model
has recently processed, which can lead to substantial savings for
applications that frequently reuse context.

The significant reduction in costs opens up major opportunities for
startups and large enterprises to explore new applications that were
previously unattainable due to budget constraints.

At the 2024 OpenAI DevDay, the pricing chart showed a significant
reduction in the cost of using AI models. Cached input tokens can save
up to 50% compared to non-cached tokens across various GPT models. The
new o1 model further highlights its advanced capabilities.

This also involves structured prompts, where caching hits are only
possible when the prefixes of prompts match exactly. To take full
advantage of caching, it's necessary to place static content (such as
instructions and examples) at the beginning of the prompt and put
variable content (such as user-specific information) at the end. This
also applies to images and tools, which must remain consistent between
requests.

### Model Distillation: Making AI Model Training More Efficient

Perhaps the most transformative change at DevDay was the introduction of
"Model Distillation." This integrated workflow allows developers to use
outputs from advanced models like o1-preview and GPT-4o to improve the
performance of more efficient models like GPT-4o mini.

In everyday training, larger AI models (such as o1-preview and GPT-4o)
can be used to fine-tune smaller models (like GPT-4o mini). This
approach enables small companies to utilise capabilities similar to
advanced models without incurring the same computational costs.

It bridges the longstanding gap in the AI industry between cutting-edge,
resource-intensive systems and more accessible but less powerful
systems. For instance, a small health tech startup developing an AI
diagnostic tool for rural clinics can use model distillation to train a
compact model that runs on a standard laptop or tablet, capturing much
of the diagnostic capability of larger models. This brings advanced AI
capabilities to resource-limited settings, potentially improving
healthcare outcomes in underserved areas.

### Visual Fine-Tuning: The New Frontier of Visual AI

Another major update is the introduction of visual fine-tuning for
OpenAI's latest large language model, GPT-4o. This feature allows
developers to customise the model's visual understanding using both
images and text.

The impact of this update is profound, with potential applications in
areas such as autonomous vehicles, medical imaging, and visual search
functionalities. According to OpenAI, Grab, a leading food delivery and
ride-hailing company in Southeast Asia, has utilised this technology to
improve its mapping services. Reportedly, with only 100 examples, Grab
increased lane count accuracy by 20% and speed sign recognition accuracy
by 13%.

This real-world application demonstrates the potential of visual
fine-tuning, showing that using small batches of visual training data
can significantly enhance AI services across industries.

**Gathering information in a "pure visual" manner through "seeing" will
greatly expand future application scenarios.** Combined with "real-time
voice interaction," OpenAI's advances in information processing and
interaction create possibilities for a new wave of applications.

Envisioned scenarios include assisting people with visual or hearing
impairments, recognising emergencies and hazards through visual
identification, and calling for help automatically.

### New Playground Features: Building a Sustainable AI Ecosystem

More explanations have been provided on the structured framework for
prompts and other usages. Parts of this information were allegedly
leaked a day earlier, including the so-called "System Prompts."

The key elements include: "Understand the task: Grasp the main
objectives, goals, requirements, constraints, and expected outputs."

- **Minimal Changes**: If an existing prompt is provided, only make
 simple improvements if possible. For more complex prompts, enhance
 clarity and add missing elements without changing the original
 structure.
- **Reasoning Before Conclusion**: Encourage reasoning steps before
 arriving at any conclusion. Attention! If the user provides an example
 where reasoning follows, REVERSE the order! Never begin examples with
 a conclusion!
- **Reasoning Order**: Highlight the reasoning parts of the prompt and
 conclusion sections (specifically named fields). For each, determine
 the ORDER in which to do so and whether a reversal is needed.
- **Conclusion, Classification, or Outcome**: These should always
 come last.

### Conclusion

OpenAI's Canvas and recent DevDay innovations represent a major step
forward in AI collaboration, offering developers powerful tools like the
Realtime API, Prompt Cache, Model Distillation, Visual Fine-Tuning, and
Playground features. These advancements make AI more accessible, enhance
productivity in writing and coding, and support industries like
healthcare and assistive technology. By bridging human creativity with
AI efficiency, OpenAI aims to foster a sustainable ecosystem that
empowers developers and creates real-world impact.

### References

- "Introducing Canvas." *Openai.com*, 2024,
 openai.com/index/introducing-canvas/.
- "X.com." *X (Formerly Twitter)*, 2024,
 x.com/karinanguyen\_/status/1841889811931791642. Accessed 12
 Oct. 2024.

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a140e4d65486){width="1"
height="1"}

#### Let's explore ChatGPT's Advanced Voice Mode

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*Z5ZUW4ye8PpT9IIV.png" />
<figcaption>Created using DALLE on October 4, 2024, using prompt
“ChatGPT’s Advanced Voice Mode”</figcaption>
</figure>

### Author

- [Wendi Fan](https://www.linkedin.com/in/wendi-fan-265996310/)
 (**ORCID: 0000--0003--0284--9166**)

### Introduction

In today's fast-paced world, where convenience and efficiency are
paramount, OpenAI's latest addition to ChatGPT, the **Advanced Voice
Mode**, sets a new standard for human-computer interaction.

Simply, ChatGPT's Advanced Voice Mode is a powerful new feature that
enables real-time, natural voice interactions. Unlike the previous
version, it natively understands speech without needing to transcribe it
into text first. This reduces delays and allows for a more fluid and
authentic conversation. Users no longer need to worry about speaking
loudly or clearly, as the advanced model can capture nuances and respond
more naturally. The experience is faster, more accurate, and feels more
like talking to a real person, enhancing user engagement and
satisfaction.

### The Technology Behind It

Two major technologies make **Advanced Voice Mode** possible:

#### 1. Automatic Speech Recognition (ASR) {#automatic-speech-recognition-asr}

At the core of Advanced Voice Mode is **Automatic Speech Recognition
(ASR)**, which directly processes your voice to understand your words.
This fast, accurate system adapts to various accents and speaking
speeds, ensuring a smooth interaction.

#### 2. Text-to-Speech (TTS) {#text-to-speech-tts}

Once your query is processed, **Text-to-Speech (TTS)** generates
natural-sounding responses. With multiple voice options available, the
voices are expressive and clear, enhancing the overall user experience.

Together, these technologies enable fluid, lifelike conversations,
almost as if you're interacting with a real assistant.

### Why OpenAI Improved Voice Mode

The old ChatGPT Voice Mode had several drawbacks that made interactions
less smooth and enjoyable. First, it involved multiple steps: your
speech was transcribed into text, processed by the language model, and
then converted back into speech. This caused delays, increasing the
chance of misunderstandings --- like missing nuances in tone or speech
clarity. Users had to speak loudly and avoid pauses to prevent
interruptions. These extra steps created a somewhat rigid and unnatural
experience, as users had to adapt to the limitations of the model
instead of having a relaxed, flowing conversation.

In contrast, the new Advanced Voice Mode natively understands speech,
removing these extra steps. This creates a smoother, more fluid
interaction, making conversations feel natural and responsive. The
improvement significantly reduces latency and enhances the overall
conversational flow, making the experience much more relaxed and
intuitive for users.

### How to Start an Advanced Mode Voice Conversation?

To start a voice conversation, we can only use the feature on mobile
devices since **Advanced Voice Mode** is not yet available on desktop.
Additionally, it's exclusive to ChatGPT subscribers.

Select the Voice icon in the bottom-right corner:

<figure>
<img
src="https://cdn-images-1.medium.com/max/550/0*ARsbouyLd19CTsUy.png" />
<figcaption>ChatGPT Voice Input Button</figcaption>
</figure>

When you begin an **advanced voice** chat, you'll see a screen with a
blue orb:

<figure>
<img
src="https://cdn-images-1.medium.com/max/303/0*2akCibVTQ8-1E5TV.jpg" />
<figcaption>User interface of ChatGPT advanced voice mode</figcaption>
</figure>

Please note that conversations using **standard voice** have a black
circle in the centre.

<figure>
<img
src="https://cdn-images-1.medium.com/max/306/0*DGPp6vaSdulclYix.jpg" />
<figcaption>User interface of old ChatGPT voice chat</figcaption>
</figure>

Once everything is set up, you're ready to start using the new voice
chat. If it's your first time using **advanced voice**, you'll be
prompted to choose a voice. You can change your voice anytime in the
settings, or within the voice mode using the customisation menu in the
top-right corner.

### What Sets Advanced Voice Mode Apart from Previous Versions?

- **Voice Variety and Realism**: The new voice mode features five new
 voices, all created in collaboration with professional voice actors.
 These voices offer a more human-like, natural sound compared to the
 old version. For example, users can now choose from a range of voices
 that vary in tone and accent.

There are nine lifelike output voices for ChatGPT, each with its own
distinct tone and character:

- Arbor --- Easygoing and versatile
- Breeze --- Animated and earnest
- Cove --- Composed and direct
- Ember --- Confident and optimistic
- Juniper --- Open and upbeat
- Maple --- Cheerful and candid
- Sol --- Savvy and relaxed
- Spruce --- Calm and affirming
- Vale --- Bright and inquisitive

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*2yYUWnqNuDJsBrwr.png" />
<figcaption>User interface of old ChatGPT voice chat</figcaption>
</figure>

These voices sound highly realistic and can even mimic the subtle noises
or pauses that naturally occur in human speech. This significantly
enhances the user experience, making interactions more lifelike and
immersive.

- **Improved Accent Support**: The new version includes better support
 for various accents, allowing for more localised and natural-sounding
 speech. For instance, it can better handle regional accents compared
 to the older version, which was less adaptive.

--- The new voice mode does a better job of mimicking different accents,
such as Italian-accented or Russian-accented English. I was pleasantly
surprised to discover that ChatGPT can also imitate regional accents in
other languages. For example, in Chinese, it can mimic accents like
Beijing, Henan, or Sichuan dialects. This versatility further enhances
the realism and personalisation of the user experience.

- **Multilingual Capability**: The new voice mode supports over 50
 languages, such as saying "Sorry I'm late" in different languages. The
 old version had limited multilingual support, making this a
 significant improvement.
- **Smoother Conversations**: The new text-to-speech engine offers
 smoother, more fluid conversations. Compared to the older version, the
 transitions between sentences and phrases are now more natural and
 less robotic.

--- The old voice chat was slow to respond, especially with complex
questions, requiring a long wait for answers. In contrast, the new voice
mode offers almost real-time responses and allows you to interrupt at
any moment to ask new questions. The interaction feels much smoother and
more seamless, greatly enhancing the overall user experience. With the
new voice chat, users no longer need to speak loudly like they did with
the old version. Even if you speak softly, the new version can
accurately recognise your voice, making the experience more convenient
and natural.

### Enhancing User Experience with Advanced Voice Mode

So, how does **Advanced Voice Mode** improve the user experience? Here
are some key enhancements:

#### 1. Increased Productivity {#increased-productivity}

For busy users, this mode allows you to perform tasks hands-free, like
requesting updates, dictating emails, or scheduling meetings without
pausing to type. It's perfect for multitasking.

#### 2. Greater Accessibility {#greater-accessibility}

The feature opens up new ways for users with disabilities to interact
with ChatGPT, making it easier to navigate without relying on screens or
keyboards.

#### 3. Engaging Learning and Entertainment {#engaging-learning-and-entertainment}

Whether practising languages or asking complex questions, **Advanced
Voice Mode** brings more immersion and ease to learning and
entertainment.

### Drawbacks of New Voice Chat

Here are some potential drawbacks of the new Advanced Voice Mode:

1. **Usage Limits**: For Plus and Team users, the **Advanced Voice
 Mode** has a daily time limit, which may change. Users receive
 notifications as they near the limit, with a final reminder when
 they have 15 minutes left for the day. Free users can access a
 monthly preview to try the feature. It's a bit disappointing that
 paid users only have limited minutes of usage per day for **Advanced
 Voice Mode**.

<figure>
<img
src="https://cdn-images-1.medium.com/max/462/0*Mq6J-7F5m6cmFa6o.jpg" />
<figcaption>Reach the limit of data</figcaption>
</figure>

**2. Pronunciation Issues**: Although it supports multiple languages,
some of the accents or pronunciations in non-English languages can sound
unnatural or inaccurate. Although the English pronunciation is
excellent, when it comes to other languages like Chinese, I noticed that
a few words sound somewhat unnatural. It's relatively easy to tell that
the voice isn't completely human in those cases.

**3. Separate Window for Advanced Mode**: You can't use **Advanced Voice
Mode** directly in the chat window; instead, you need to open a new chat
window to start using it.

### Conclusion

In summary, ChatGPT's **Advanced Voice Mode** offers a significant leap
forward in voice interaction, providing smoother, more realistic
conversations with enhanced accent and multilingual support. It greatly
improves productivity, accessibility, and overall user engagement.
However, the feature does come with limitations such as daily usage caps
for subscribers, some pronunciation issues in non-English languages, and
the need to use a separate window for voice interactions. Despite these
drawbacks, the improvements in user experience make it a valuable tool
for efficient and immersive communication.

### References

- Dan Shipper. "Review: ChatGPT's New Advanced Voice Mode". *Every*, 9
 Aug. 2024. [Read
 me](https://every.to/chain-of-thought/review-chatgpt-s-new-advanced-voice-mode)).
 Accessed on 10 Oct 2024.
- *OpenAI Team --- Voice mode FAQ*
 [Read me](https://help.openai.com/en/articles/8400625-voice-mode-faq).
- Jordan Novet. "OpenAI just launched advanced voice mode for audio
 chats with ChatGPT. Here's how to use it". *CNBC*, 25 Sep. 2024. [Read
 me](https://www.cnbc.com/2024/09/24/how-to-use-chatgpt-advanced-voice-mode-from-openai.html).
 Accessed on 10 Oct 2024.
- Reece Rogers. "I Used ChatGPT's Advanced Voice Mode. It's Fun, and
 Just a Bit Creepy". *WIRED*, 13 Aug 2024. [Read
 me](https://www.wired.com/story/chatgpt-advanced-voice-mode-first-impressions/).
 Accessed on 10 Oct 2024.

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=69467a559783){width="1"
height="1"}