Australian National University

Qingqin Fang

Transformative Advances in Language Models through External Knowledge Integration

Author: Qingqin Fang (

 ORCID:

0009–0003–5348–4264)

 Introduction

In the dynamic field of natural language processing, the integration of external knowledge has emerged as a pivotal strategy for enhancing the performance of language models.

Deeper understanding of In-Context Retrieval-Augmented Language Models

Wendi Fan

Let’s explore GR-2 together. Author Wendi Fan (

 ORCID: 0000–0003–0284–9166

) Introduction GR-2 is a cutting-edge generative model designed for versatile and generalisable robot manipulation, developed by the Robotics Research Team at ByteDance Research. It represents a significant step forward in robotics, enabling robots to execute a wide range of manipulation tasks in various environments.

GR-2: A Generative Video-Language-Action Model for Robot Manipulation

Vaibhav Khobragade

From Naive to Modular: Tracing the Evolution of Retrieval-Augmented Generation

Author · Vaibhav Khobragade (

 ORCID:

0009–0009–8807–5982) Introduction Large Language Models (LLMs) have achieved remarkable success.

Three Paradigms of RAG

Wenyi Pi

Exploring innovative Strategies in Combating Misinformation with Enhanced Multimodal Understanding Author Wenyi Pi (

 ORCID

: 0009–0002–2884–2771) Introduction Misinformation refers to false or inaccurate information that is often given to someone in a deliberate attempt to make them believe something that is not true. This has a significantly negative impact on public health, political stability and social trust and harmony.

Multimodal Large Language Models for Misinformation Detection and Reasoning

Amanda Kau

Techniques to integrate Knowledge Graphs into Language Models

 Author

Amanda Kau

 (ORCID:


 0009–0004–4949–9284


 )

Introduction Both knowledge graphs (KGs) and pre-trained language models (PLMs) have gained popularity due to their ability to comprehend world knowledge and their broad applicability.

A Deep Dive Into Knowledge Graph Enhanced Pre-trained Language Models

Dhruv Gupta

Understanding the Power and Applications of Natural Language Processing

 Author

Dhruv Gupta (

 ORCID:

0009–0004–7109–5403) Introduction We are living in the era of generative AI. In an era where you can ask AI models almost anything, they will most certainly have an answer to the query. With the increased computational power and the amount of textual data, these models are bound to improve their performance.

NLU vs. NLG: Unveiling the Two Sides of Natural Language Processing

Zijian Yang

Le Chat by Mistral: Access ChatGPT-Like Features, Image Generation, Canvas, Artifact, and More for Free


 Discover Le Chat: Free ChatGPT Alternatives with Canvas, Internet Search, and Flux.1 Image Generation


 Author

Zijian Yang (

 ORCID:

0009–0006–8301–7634)

 Introduction

Imagine accessing premium ChatGPT features — Canvas, internet search, and PDF

Le Chat by Mistral: Access ChatGPT-Like Features, Image Generation, Canvas, Artifact, and More for…

Enhancing Data Interactivity with LLMs and Neo4j Knowledge Graphs Author Wenyi Pi (

 ORCID

: 0009–0002–2884–2771) Introduction Since OpenAI launched ChatGPT, a

 large language model (LLM)

based chatbot, in 2023, it has set off a technological wave.

The integration of large language models (LLMs) with Neo4j-based knowledge graphs

Xuzeng He

Latest findings in multiple research directions for tackling reasoning and common sense challenges Author: Xuzeng He (

 ORCID:

0009–0005–7317–7426) Knowledge Graphs, such as Wikidata, contain rich relational information between entities and have been widely used as a structured format for storing and representing relational information.

Large Language Models and Knowledge Graphs: Ways to combine them

Swinburne University of Technology

Nakul Nambiar

Aishwarya Nambissan

A novel approach to improving the efficiency of text search in graph databases utilizing Neo4j, OpenAI, and Typesense. Authors  Nakul Nambiar (ORCID: 0009–0009–9720–9233) Aishwarya Nambissan (ORCID: 0009–0003–3823–6609)  The ability to use cutting-edge tools and frameworks is essential for staying ahead in the ever-changing field of technology.

Typesense and Neo4j in a hybrid information retrieval solution

Exploring the OpenAlex Data Structure and Visualization Author: Qingqin Fang (

 ORCID:

0009–0003–5348–4264)

 Introduction to OpenAlex

In today’s world, the realm of research papers is brimming with countless hot topics, and the sheer volume of publications can be overwhelming.

Unveiling Research Trends through OpenAlex Visualization

Neo4j APOC Library Use Case

 Author

Wenyi Pi (

 ORCID:

0009–0002–2884–277)

 Introduction

In the realm of Neo4j, the APOC (Awesome Procedures on Cypher) library stands as a powerful tool. Previously, We have talked about the importance of APOC in optimising Cypher queries and improving query efficiency in our article Exploring Methods of Cypher Query Optimisations.

Introduction to APOC: Enhancing Neo4j Capabilities

Unlocking the Future of AI: The Transformative Journey of Large Language Models Author · Vaibhav Khobragade (

 ORCID:

0009–0009–8807–5982)

 Introduction

Human language development is innate and evolves throughout life. Machines lack this ability to evolve without advanced AI algorithms.

The Journey of Large Language Models: Evolution, Application, and Limitations

Unlocking the Power of Questions — A deep dive into Question Answering Systems

Author:

 Amanda Kau (ORCID:


 0009–0004–4949–9284


 )

Virtual assistants have popped up on numerous websites over the years.

Harnessing The Power of Knowledge Graphs in Question Answering

Dhruv Gupta

Improving the performance of Large Language Models Author  Dhruv Gupta (ORCID: 0009-0004-7109-5403)  ChatGPT, which first came out in late 2022, took the world by storm. Since then, various LLM models and LLM based products such as Meta’s Llama and Google’s Gemini have emerged, demonstrating the power of LLMs.

RAG: The next big thing after LLMs?

Amir Aryani

Authors  Nakul Nambiar (ORCID: 0009-0009-9720-9233) Amir Aryani (ORCID: 0000-0002-4259-9774)  Knowledge graphs, which offer a structured representation of data and its relationships, are revolutionising how we organise and access information.

Mini RAG using Neo4j

Author  Amir Aryani (ORCID: 0000-0002-4259-9774) Introduction   In this article we look at Research Graph as an information model , and an approach to connect and capture the connections between research outputs, researchers and research activities. We explore the metadata model, and we discuss how to capture this graph in a Neo4j Graph Database.

Research Graph 101

Aland Astudillo

How to use GROBID to extract text from PDF  Author  Aland Astudillo (ORCID: 0009-0008-8672-3168)  GROBID is a powerful and useful tool based on machine learning that can extract text information from PDF files and other files to a structured format. One of the key challenges in knowledge mining from academic articles is reading the content of PDF files.

How to use GROBID

Unlocking the power of language models: A deep dive into BERT


 Author:


 Dhruv Gupta (ORCID:


 0009–0004–7109–5403


 )

Clive Humby, in 2006 rightly said, “Data is the new oil”. With data being present everywhere, it has never been more valuable.

Language Models: Deep Dive into BERT

An introduction to tools supporting PubMed Author: Xuzeng He (

 ORCID:

0009–0005–7317–7426) PubMed is a free Web literature search service developed and maintained by the National Center for Biotechnology Information (

 NCBI

), and it is also a part of NCBI’s Entrez retrieval system that provides access to a diverse set of 38 databases.

Enhancing PubMed: from a Medical Database to beyond

Exploring the potentials and limitations of Vision Language Models

Author:

 Amanda Kau (ORCID:


 0009–0004–4949–9284


 )

The human brain is more extraordinary than any machine we could build. From an early age, many of us gain the ability to comprehend what our eyes tell us and articulate it. Furthermore, we combine evidence from all our senses to reason.

How Much Can Vision Language Models Really “See”?

How to efficiently retrieve information for different applications

Author Wenyi Pi (ORCID: 0009-0002-2884-2771) This article aims to explore various ways in which Retrieval-Augmented Generation (RAG) can be utilised to retrieve information and generate responses effectively within the dialogue system. The rationale behind utilising RAG as well as potential ways in which it can be employed effectively will be covered.

Efficient Information Retrieval and Response Generation with Retrieval-Augmented Generation (RAG)

Addressing Misinformation, Bias, and Privacy Concerns

Author:

 Wenyi Pi (ORCID:


 0009–0002–2884–2771


 )

Recently, the spread of false or misinformation has become an important concern for people all over the world, especially in the era of big data and multimedia, where people are faced with a flood of information that makes it difficult to determine its authenticity.

The Ethical and Social Impact of Artificial Intelligence

Science in the age of large language models

Generative AI entails a credit–blame asymmetry

Exploring AI’s Ethical Terrain: Addressing Bias, Security, and Beyond

Author: Vaibhav Khobragade (

 ORCID:

0009–0009–8807–5982) Large language models (LLMs) like OpenAI’s GPT-4, Meta’s LLaMA, and Google Gemini (previously called Bard) have showcased their vast capabilities, from passing bar exams and crafting articles to generating images and website code.

Ethics and AI: Confronting the Challenges Ahead

The AI Helper Turning Mountains of Data into Bite-Sized Instructions

Author Aland Astudillo (ORCID: 0009-0008-8672-3168) LLMs have been changing the way the entire world deals with problems and day-by-day tasks. To make them better for specific applications, they need huge amounts of data and complex and expensive approaches to training them.

What is LLMLingua?

Zhuochen Wu

Authors:  Nakul Nambiar (ORCID: 0009-0009-9720-9233) Zhuochen Wu (ORCID: 0009-0000-5642-5348)  Research Graph is a structured representation of research objects that captures information about entities and the relationships between Researcher, Organisation, Publication, Grant and Research Data.

How to use GPT API to export a research graph from PDF publications

Tools and Platform for Integration of Knowledge Graph with RAG pipelines.

Authors Aland Astudillo (ORCID: 0009-0008-8672-3168) Aishwarya Nambissan (ORCID: 0009-0003-3823-6609) Many users of chatbots such as ChatGPT, have encountered the problem of receiving inappropriate or incompatible responses. There are several reasons why this might happen.

Unveiling the Synergy: Retrieval Augmented Generation (RAG) Meets Knowledge Graphs

Amir Aryani

A brief overview of different types of clustering techniques and their algorithms. Authors Aishwarya Nambissan (ORCID: 0009-0003-3823-6609) Amir Aryani (ORCID: 0000-0002-4259-9774)

 Background

Clustering is a fascinating technique used in machine learning, where patterns or data points are grouped based on their similarities. It’s like finding hidden connections among different data points without predefined labels.

19 Clustering Techniques

Unlocking the power of knowledge graphs in research catalogues: A deep dive into OpenAlex


 Author:


 Dhruv Gupta (ORCID:


 0009–0004–7109–5403


 )

Clive Humby, in 2006 rightly said, “Data is the new oil”. With data being present everywhere, it has never been more valuable.

Knowledge Graphs: Deep Dive into OpenAlex

Improving the performance and application of Large Language Models

Author Amanda Kau (ORCID: 0009-0004-4949-9284) Large language models (LLMs) like GPT-4, the engine of products like ChatGPT, have taken centre stage in recent years due to their astonishing capabilities. Yet, they are far from perfect.

Understanding Retrieval Pitfalls: Challenges Faced by Retrieval Augmented Generation (RAG) models

Yunzhong Zhang

Efficient creation of a stoplight report with data dashboard images

Author: Yunzhong Zhang (ORCID: 0009–0002–8177–419X) Comparing data dashboards is crucial for understanding trends and performance differences. Traditionally, this task required manual effort, which was slow and sometimes inaccurate. Now, thanks to OpenAI’s GPT-4 with Vision (GPT-4V), we are able to automate and improve this process.

How to use GPT-4V For Stoplight Report

Author Dhruv Gupta (

 ORCID

: 0009–0004–7109–5403) Introduction Large Language Models (LLMs) have become the new face of Natural language processing (NLP). With their generative power and ability to comprehend human language, the human reliance on these models is increasing every day. However, the LLMs have been known to hallucinate and thus produce wrong outputs.

Prompt engineering: A Way to Smartly Use AI

Recent Advances in Using Machine Learning with Graphs — Part 2 Latest findings in multiple research directions for handling graph construction and network security issues Author · Xuzeng He (

 ORCID:

0009–0005–7317–7426) Introduction A graph, in short, is a description of items linked by relations, where the items of a graph are called nodes (or vertices) and their relations are called edges (or links). Examples of

Recent Advances in Using Machine Learning with Graphs — Part 2

Bridging Human Perception and AI’s Future: The Convergence of Visual Understanding and Semantic Networks Author · Vaibhav Khobragade (

 ORCID:

0009–0009–8807–5982)

 Introduction

The fusion of Vision-Language Models (

 VLMs

), Generative Models, and Knowledge Graphs (

 KGs

) is reshaping how artificial intelligence (AI) understands and interacts with the world.

Tuning Vision-Language Models and Generative Models with Knowledge Graph

Using intelligence to use artificial Intelligence: A deep dive into Prompt Engineering


 Author


 Dhruv Gupta (ORCID:


 0009–0004–7109–5403


 )

Introduction Large Language Models (LLMs) have become the new normal in the field of Natural Language Processing (NLP). With their improved performance and generative power, people around the world are relying on it for

Prompt Engineering

An Overview of Constructing a Knowledge Graph Author · Qingqin Fang (

 ORCID:

0009–0003–5348–4264) 1. Introduction 1.1 What is a Knowledge Graph Knowledge Graphs are structured semantic knowledge bases used to rapidly describe concepts and their relationships in the physical world.

Unlocking Intelligence: The Journey from Data to Knowledge Graph

Boosting Performance for Knowledge Graphs with Neo4j APOC Library

 Author

Wenyi Pi (ORCID: 0009–0002–2884–2771) Introduction A

 knowledge graph

(graph database) captures information about main entities in a domain and the relationships between them. It was an augmented feature store for connected data which gave access to compute, access and operationalise structure features.

Exploring Methods of Cypher Query Optimisations

Aditya Iyengar

How AI is Transforming Medicine and Improving Lives
 
 AI in the Healthcare Domain
 


 Author

Aditya Iyengar (ORCID: 0009–0005–1959–9724)

 Introduction

The rapid integration of Artificial Intelligence (AI) into healthcare is no longer just a futuristic vision — it is happening now and at an extraordinary pace.

The Future of Healthcare

The Most Recent Cutting-Edge Models


 Author

Aditya Iyengar (ORCID: 0009–0005–1959–9724)

 Introduction

The past 6 months marked significant technological leaps in the field of artificial intelligence (AI), especially in the realm of image generation. In this six-month window, multiple innovations redefined the capabilities of text-to-image, video generation, and multimodal AI systems.

The New Era of AI Image Generation

Mingrui Gao

Working Examples


 Author

Mingrui Gao (

 ORCID

: 0009–0005–7271–2677)

 Introduction

Machine learning has revolutionised countless industries, but deploying ML models remains a significant challenge for many data scientists and developers.

What is Cog?

Let’s explore MLflow together. Author Wendi Fan (

 ORCID: 0000–0003–0284–9166

) Introduction MLflow is an open-source platform designed to manage the entire lifecycle of machine learning projects. It helps developers and data scientists streamline their workflows by tracking experiments, managing models, and deploying them efficiently.

MLflow Beginner’s Guide: How to Get Started with MLflow

Tarun Krishnan

Unleashing FLUX1.1 with the BFL API


 Author

Tarun Krishnan (

 ORCID

: 0009–0006–6647–127X)

 Introduction

With the growing demand for

 image generation models

in various applications, the need for

 APIs

that seamlessly integrate these models into developers’ workflows has skyrocketed.

The Fast Lane of Image Generation

Tohfa Siddika

The Godfather of AI Author Tohfa Siddika Barbhuiya (

 ORCID

: 0009–0007–2976–4601) In a monumental moment for the world of Artificial Intelligence (AI), Geoffrey Hinton, known as the “Godfather of AI,” has been awarded the Nobel Prize for his pioneering work in neural networks and deep learning. He has been awarded the 2024 Nobel Prize in physics by the Royal Swedish Academy of Sciences.

Geoffrey Hinton Wins Nobel Prize

Evaluating the Role of Self-Reflection in AI: Innovation or Overcomplication?

Author Tohfa Siddika Barbhuiya (

 ORCID

: 0009–0007–2976–4601) Introduction Artificial Intelligence (AI) has made remarkable strides over the past decade, with new advancements pushing the boundaries of what machines can achieve. One of the latest developments in the field is the introduction of “reflection” within AI models.

The Idea of Reflection Models

OpenAI Canvas and DevDay: Advancing AI Collaboration with Voice Realtime API, Model Distillation, and More Explore OpenAI’s latest tools and features — Canvas, Realtime API, Prompt Cache, Visual Fine-Tuning Author Zijian Yang (

 ORCID:

0009–0006–8301–7634) Introduction OpenAI recently introduced a new feature called Canvas, which is designed to provide ChatGPT users with a more powerful writing and coding collaboration

OpenAI Canvas and DevDay: Advancing AI Collaboration with Voice Realtime API, Model Distillation…

Working Examples Author Mingrui Gao (

 ORCID

: 0009–0005–7271–2677) Introduction In the rapidly evolving landscape of artificial intelligence, Large Language Models (LLMs) have emerged as powerful tools capable of understanding and generating human-like text. While cloud-based services offer convenient access to these models, there’s a growing demand for local, hands-on experimentation.

What is LM Studio?

A novel compression technique ensuring comparable performance with 70% less parameters

Author Amanda Kau (

 ORCID

: 0009–0004–4949–9284) Introduction The sizes of large language models (LLMs) have been steadily increasing over the last few years.

CompactifAI: Large Language Models Don’t Actually Have To Be Large

Let’s explore ChatGPT’s Advanced Voice Mode Author Wendi Fan (

 ORCID: 0000–0003–0284–9166

) Introduction In today’s fast-paced world, where convenience and efficiency are paramount, OpenAI’s latest addition to ChatGPT, the

 Advanced Voice Mode

, sets a new standard for human-computer interaction.

Unlocking the Power of ChatGPT’s Advanced Voice Mode

Automated Knowledge Graph Construction with Large Language Models — Part 2

 Harvesting the Power and Knowledge of Large Language Models

Author Amanda Kau (

 ORCID

:

 0009–0004–4949–9284

) Introduction Knowledge graphs (KGs) are a structured representation of data in a graphical format, in which entities are represented by nodes and are connected by edges representing relationships

Automated Knowledge Graph Construction with Large Language Models — Part 2

Understanding the Evolutionary Journey of LLMs Author Wenyi Pi (

 ORCID

: 0009–0002–2884–2771) Introduction When we talk about large language models (LLMs), we are actually referring to a type of advanced software that can communicate in a human-like manner. These models have the amazing ability to understand complex contexts and generate content that is coherent and has a human feel.

Brief Introduction to the History of Large Language Models (LLMs)

An improvement architecture superior to the Transformer, proposed by Meta Author · Qingqin Fang (

 ORCID:

0009–0003–5348–4264) Introduction Recently, researchers from Meta and the University of Southern California have introduced a model called Megalodon. They claim that this model can expand the context window of language models to handle millions of tokens without overwhelming your memory.

The longer the context, the better? Unlimited Context Length in Megalodon

Attention mechanism not getting enough attention Author Dhruv Gupta (

 ORCID

: 0009–0004–7109–5403) Introduction As discussed in this article, RNNs were incapable of learning long-term dependencies. To solve this issue both LSTMs and GRUs were introduced. However, even though LSTMs and GRUs did a fairly decent job for textual data they did not perform well.

Transformers Models in NLP

The Three Oldest Pillars of NLP

Author Dhruv Gupta (

 ORCID

: 0009–0004–7109–5403) Introduction Natural Language Processing (NLP) has almost become synonymous with Large Language Models (LLMs), Generative AI, and fancy chatbots. With the ever-increasing amount of textual data and exponential growth in computational knowledge, these models are improving every day.

RNNs vs GRUs vs LSTMs

Large Language Models for Fake News Generation and Detection Author Amanda Kau (

 ORCID

: 0009–0004–4949–9284) Introduction In recent years, fake news has become an increasing concern for many, and for good reason. Newspapers, which we once trusted to deliver credible news through accountable journalists, are vanishing en masse along with their writers.

Are Large Language Models Our Allies or Enemies in the Fight Against Fake News?

A Unified and Collaborative Framework for LLM Author · Qingqin Fang (

 ORCID:

0009–0003–5348–4264) Introduction In today’s rapidly evolving field of artificial intelligence, large language models (LLMs) are demonstrating unprecedented potential. Particularly, the Retrieval-Augmented Generation (RAG) architecture has become a hot topic in AI technology due to its unique technical capabilities.

RAG 2.0 is Coming?

Exploring the Potential of Temporal Feature-Logic Embedding (TFLEX) in Complex Query Resolution

Author · Vaibhav Khobragade (

 ORCID:

0009–0009–8807–5982)

 Introduction

Artificial intelligence (AI) and knowledge representation in the field of temporal knowledge graphs are rapidly gaining interest.

Harnessing Temporal Dynamics: Advanced Reasoning using Temporal Knowledge Graphs

Integrating temporal data into static knowledge graphs

 Author

Amanda Kau

 (ORCID:


 0009–0004–4949–9284


 )

Introduction Knowledge graphs (KGs) have proven to be an effective method of data representation that is increasingly popular. In KGs, entities and concepts are represented as nodes, while the relationships between nodes are depicted as edges.

Dynamic Knowledge Graphs: A Next Step For Data Representation?

Understanding the Balance between Internal Knowledge and External Sources

Author Qingqin Fang (

 ORCID:

0009–0003–5348–4264) Introduction Previous research often emphasized the limitations of LLM’s information acquisition pathways, focusing on enhancing its capabilities in this regard.

When RAG and LLM Conflict: Who Will AI Listen to?

Exploring the Boundaries of Creativity and Responsibility in the Age of AI-Driven Media

 Author

Vaibhav Khobragade (

 ORCID:

0009–0009–8807–5982)

 Introduction

In 2024, the discipline of Generative AI takes a big step forward with the launch of revolutionary models that convert text into dynamic films, altering the landscape of digital content creation.

Generative AI’s Leap into Video in 2024 and its Ethical Horizon

Author  Amir Aryani: (ORCID: 0000-0002-4259-9774) Definition   A research collaboration network is a group of researchers, and practitioners, or both, working together on joint research activities. These networks often span across disciplines, geographic boundaries, and sectors, enabling participants to share resources, expertise, and data to address common research goals more effectively than they could individually.

Research Collaboration Network

Understanding Sequential Data Modelling with Keras for Time Series Prediction

 Author

Wenyi Pi (

 ORCID

: 0009–0002–2884–2771) Introduction Recurrent Neural Networks (RNNs) are a special type of neural networks that are suitable for learning representations of sequential data like text in Natural Language Processing (NLP). We will walk through a complete example of using RNNs for time series prediction, covering

Beginner’s Guide to Recurrent Neural Networks (RNNs) with Keras

Incorporating Knowledge Graphs to explain reasoning processes


 Author

Amanda Kau (

 ORCID:

0009–0004–4949–9284) Introduction Large language models (LLMs) like GPT-4 possess remarkable language abilities, allowing them to function as chatbots, translators, and much more.

Combining Knowledge Graphs With Language Models for Interpretability

Latest findings in multiple research directions for handling graph prediction and optimization Author · Xuzeng He (

 ORCID:

0009–0005–7317–7426) A graph, in short, is a description of items linked by relations, where the items of a graph are called

 nodes

(or vertices) and their relations are called

 edges

(or links). Examples of graphs can include social networks (e.g. Instagram) or knowledge

Recent Advances in using Machine Learning with Graphs

Prompt Engineering — Part 2


 Using intelligence to use artificial Intelligence: A deep dive into Prompt Engineering

Author Dhruv Gupta

 (ORCID:

0009–0004–7109–5403

 )

Introduction In the previous article we discussed what prompt engineering and some of the techniques used for prompt engineering.

Prompt Engineering — Part 2

Using Mistral for Data tagging Author · Xuzeng He (

 ORCID:

0009–0005–7317–7426) Introduction Data tagging, in simple terms, is the process of assigning labels or tags to your data so that they are easier to retrieve or analyse.

How to use Large Language Models to tag your data: A complete tutorial

Ministral — The World’s Best Edge Language Model
 


 Revolutionising On-Device AI: High-Performance, Low-Resource Language Models for Edge Computing


 Author

Tarun Krishnan (

 ORCID

: 0009–0006–6647–127X)

 Introduction

In recent years, there’s been a notable shift in the AI landscape, with many companies moving away from developing massive,

Ministral — The World’s Best Edge Language Model

Author: Vaibhav Khobragade (

 ORCID:

0009–0009–8807–5982) Introduction Large language models (LLMs) are becoming increasingly popular in natural language processing for their superior competence in various applications.

Enhancing Language Models: The Role of Knowledge Graph Augmentation in Overcoming LLM Challenges

An Introduction to RA-CM3, MuRAG and RACE Author  Xuzeng He (ORCID: 0009-0005-7317-7426)  Generative Artificial Intelligence (GAI) has demonstrated impressive performances in tasks such as text generation and text-to-image generation.

Recent Advances in using Retrieving Multimodal Information for Augmented Generation

Understanding Knowledge Networks From A Graph Perspective


 Author:


 Amanda Kau (ORCID:


 0009–0004–4949–9284


 )

Since 2020, over ten million scholarly articles have been published annually. To put that into perspective, say all ten million articles were released on the first day of the year.

Uncovering The Secrets Behind The Most Influential Scholarly Publications With AceMap

Latest effort in assessing the security of the code generated by large language models Author · Xuzeng He (

 ORCID:

0009–0005–7317–7426) Introduction With the surge of Large Language Models (LLMs) nowadays, there is a rising trend among developers to use Large Language Models to assist their daily code writing. Famous products include GitHub Copilot or simply ChatGPT.

Large Language Models for Code Writing: Security Assessment

Harvesting the Power and Knowledge of Large Language Models


 Author

Amanda Kau

 (ORCID:

0009–0004–4949–9284

 )

Introduction Knowledge Graphs are networks that represent data in a graphical format. The beauty of Knowledge Graphs lies in their representation of concepts, events and entities as nodes, and the relationships between them as edges.

Automated Knowledge Graph Construction with Large Language Models

Understanding how RNNs work and its applications


 Author

Wenyi Pi (

 ORCID

: 0009–0002–2884–2771) Introduction In the ever-evolving landscape of artificial intelligence (AI), bridging the gap between humans and machines has seen remarkable progress. Researchers and enthusiasts alike have tirelessly worked across numerous aspects of this field, bringing about amazing advancements.

An Introduction to Recurrent Neural Networks (RNNs)

Latest findings for the use of Knowledge Graph in the field of QA in multiple research directions Author: Xuzeng He (

 ORCID:

0009–0005–7317–7426) Question Answering (QA), the ability to interact with data using natural language questions and obtaining accurate results, has been a long-standing challenge in computer science dating back to the 1960s.

Question Answering enhanced by Knowledge Graph

Latest findings in pre-training graphs and using them for link recommendation Author · Xuzeng He (

 ORCID:

0009–0005–7317–7426) Introduction A graph, in short, is a description of items linked by relations, where the items of a graph are called nodes (or vertices) and their relations are called edges (or links). Examples of graphs can include social networks (e.g. Instagram) or knowledge graphs (e.g. Wikipedia). In Instagram

Working with Graphs: Pre-training and Application

Anthropic’s Groundbreaking AI, Claude 3.5, Uses Computers Like a Human and Becomes a Game-Changer in Automation


 Author

Zijian Yang (

 ORCID:

0009–0006–8301–7634)

 Introduction

Claude 3.5 Receives a Major Upgrade Overnight! As anticipated, Anthropic AI made a significant move this week with the launch of Claude 3.5 Haiku.

New Claude 3.5 Can Control Computer: Outsmarts o1 in Coding and Redefines Agent Capabilities

Solutions to Enhance LLM Performance in Long Contexts Author · Qingqin Fang (

 ORCID:

0009–0003–5348–4264) Introduction In the era of AI breakthroughs, large language models (LLMs) are not just advancements; they are revolutions, transforming how we interact with technology, from casual conversations with chatbots to the intricate mechanisms behind sophisticated data analysis tools.

Navigating the Long Context Conundrum: Challenges in Language Models’ Information Processing

Supervised Fine-tuning, Reinforcement Learning from Human Feedback and the latest SteerLM Author · Xuzeng He (

 ORCID:

0009–0005–7317–7426) Introduction Large Language Models (LLMs), usually trained with extensive text data, can demonstrate remarkable capabilities in handling various tasks with state-of-the-art performance. However, people nowadays typically want something more personalised instead of a general solution.

Fine-tuning Large Language Models: A Brief Introduction

How Apple’s Research Exposes AI’s Inability to Reason and OpenAI Highlights ChatGPT’s Name-Based Biases


 Author

Zijian Yang (

 ORCID:

0009–0006–8301–7634)

 Introduction

As artificial intelligence continues to evolve, the capabilities and limitations of large language models (LLMs) are under increasing scrutiny.

ChatGPT’s Name Bias and Apple’s Findings on AI’s Lack of Reasoning: Major Flaws Revealed

Working Examples


 Author

Mingrui Gao (

 ORCID

: 0009–0005–7271–2677)

 Introduction

In the fast-paced world of artificial intelligence, a new platform is gaining attention by making AI-powered content creation more accessible.

What is Glif?

Stability AI’s latest release offering New Models and Features


 Author

Tohfa Siddika Barbhuiya (

 ORCID

: 0009–0007–2976–4601)

 Stable Diffusion 3.5

by

 Stability AI

marks a major step forward in AI-driven image generation.

Stable Diffusion 3.5

Research Graph

Google’s Groundbreaking AI Model Explained

Introduction In the rapidly evolving landscape of artificial intelligence, Google continues to stand at the forefront, consistently pushing the boundaries of what these technologies can achieve. One of their latest advancements, GEMMA2, represents a significant leap in AI capabilities. Let’s delve into the details of this new AI model by Google.

What is Gemma2?

Enhancing Open-Domain Conversational Question Answering with Knowledge-Enhanced Models and Knowledge Graphs


 How knowledge-enhanced language models and knowledge graphs are advancing open-domain conversational question answering

Author:

 Wenyi Pi (ORCID:


 0009-0002-2884-2771


 )

When searching for information on the website, it is common to come across a flood of

Enhancing Open-Domain Conversational Question Answering with Knowledge-Enhanced Models and…

Refining AI Vision: How Retrieval-Augmented Generation Transforms Image Captioning in Large Language Models   Leveraging External Knowledge to Enhance the Descriptive Capabilities of AI Systems Author  Vaibhav Khobragade (ORCID: 0009–0009–8807–5982) Introduction   Large Language Models (LLMs) are artificial intelligence models that are trained on massive amounts of text data in order to generate human-like language and produce coherent

Refining AI Vision: How Retrieval-Augmented Generation Transforms Image Captioning in Large…

An Introduction to Retrieval Augmented Generation (RAG) and Knowledge Graph

Author Qingqin Fang (ORCID: 0009–0003–5348–4264)

 Introduction

Large Language Models (LLMs) have transformed the landscape of natural language processing, demonstrating exceptional proficiency in generating text that closely resembles human language.

Hallucination in Large Language Models and Two Effective Alleviation Pathways

Stories by Research Graph on Medium

#### **Exploring the Potential of Temporal Feature-Logic Embedding (TFLEX) in Complex Query Resolution**

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*H6ME71WASxb20ZcS" />
<figcaption>Weaving Time in Knowledge Graphs by ChatGPT
(DALLE2).</figcaption>
</figure>

### Author

· Vaibhav Khobragade (**ORCID:**
[0009--0009--8807--5982](https://orcid.org/0009-0009-8807-5982))

### **Introduction**

Artificial intelligence (AI) and knowledge representation in the field
of temporal knowledge graphs are rapidly gaining interest. Current
challenges include developing models and frameworks that can effectively
form reasoning mechanisms. **Temporal knowledge graphs (TKGs)** refer to
a specific group of knowledge representation systems that besides
concepts, entities, events, and relationships, capture time information
(Li et al., 2021). These graphs give so much useful material in the
processes of forecasting, organizing plans, and decision-making.

TKGs are active databases that not only drive the information but also,
the context of true incidents when they happen and that makes them
crucial in the prediction of forthcoming trends and the understanding of
complex sequences of events.

Innovative techniques termed **Temporal Feature-Logic Embedding
framework (TFLEX)** were introduced, which is purposefully built to deal
with temporal and logical issues especially to handle queries that have
elements of time information, that classic static KGs would fail to
deal with.

***Let's build a fundamental concept before we delve into TFLEX:***

### **What is a Knowledge Graph?**

A knowledge graph is a way of storing information that uses a
graph-structured model. In this model, entities are represented as nodes
and the relationship between them are represented as edges, connected to
the nodes. This structure depicts complex interrelationships
within data.

For example: An entity "Inception" is connected to an entity "Leonardo
DiCaprio" with the relationship "acted in" and to "Christopher Nolan"
entity with the relationship "directed by".

<figure>
<img src="https://cdn-images-1.medium.com/max/940/0*Xa_zd0gENuy-GKlf" />
<figcaption>Figure1: Knowledge graph (entities and relationship
example)</figcaption>
</figure>

This structure makes it easy to understand the connections between
pieces of information and to use them for reasoning, searching, and
knowledge discovery.

### What is a Temporal Knowledge Graph?

A Temporal Knowledge Graph (TKG) enhances traditional static knowledge
graphs by incorporating time data into the structure. This is achieved
by adding a timestamp to the usual triple format, resulting in elements
described as \<head, relation, tail, timestamp\>.

For instance, consider the data point \<Bill Gates, makes a visit,
India, 2024--04--23\>; these records not only the event and the entities
involved but also the exact date it occurred.

### **TFLEX**

Recent research on **Temporal Knowledge Graph Completion (TKGC)** aims
to infer new facts in TKGs that focus on temporal link prediction, which
is simply a single hop. It tries to predict if two nodes that are not
currently connected will form a single direct link in the future based
on the history of the network and its evolution over time. Some TKGC
methods like **timestamp-based transformation, dynamic embedding, and
Markov process** **model** work by performing multi-hop reasoning.
However, none of them could answer logical questions involving
conjunction, negation, and disjunction operation

For instance, find all scientists who have published their research in
the field of Artificial Intelligence (Conjunction), have never received
an award (Negation), or have collaborated with certain universities
before 2020 (Before and Disjunction). In this article, we focused on
multi-hop logical reasoning to answer the temporal complex queries. To
understand this method, we should know what temporal complex
queries are.

### **Temporal Complex Queries**

This type of query for Temporal Knowledge Graphs (TKGs) specifically
includes both entity and temporal elements. These queries are complex
because they require reasoning across multiple connections (multi-hop)
and involve both entities and the times associated with their
relationships. For example, a query might ask which events a person
attended before a certain year and who else was present at those events.
This type of query goes beyond simple fact retrieval, incorporating
logical and temporal operators to extract structured information across
time and entities as discussed above.

In a mathematical way, a temporal complex query can be represented as:

<figure>
<img src="https://cdn-images-1.medium.com/max/940/0*L6jPwbDy21yoABqZ" />
</figure>

The above query will be discussed in more detail in the paragraphs
that follow.

### **Data Generation Process**

Researchers formulated the above temporal complex queries that involve
both entities and their temporal relationships and then applied these
queries to three different popular TKGs to generate three datasets
(ICEWS14, ICEWS05--15, and GDELT-500) for further training and testing
this model.

Multi-hop logical reasoning over knowledge graphs is a complex process
that involves answering queries by traversing multiple relationships (or
"hops") between entities in a knowledge graph. In simpler terms, it's
like solving a puzzle where each piece (or hop) brings you closer to the
complete answer by following a trail of interconnected data points.

<figure>
<img src="https://cdn-images-1.medium.com/max/940/0*BCcPSrLzstsI3hVF" />
<figcaption>Figure 2: An example of a multi-hop temporal complex query
and its processing network is as follows: “Which countries did Xi
Jinping visit but Barack Obama did not visit during François Hollande’s
presidency of France?” Entity sets (blue circles), timestamp sets (green
triangles), entity set projections (blue arrows), and logical operators
(red rectangles) are represented in the computation graph. (Lin et al.,
2022) Article link: <a
href="https://arxiv.org/abs/2205.14307">https://arxiv.org/abs/2205.14307</a></figcaption>
</figure>

1. **Entity and Timestamp Representation**:

- **Entity Nodes (Circles)**: Entities such as "Xi Jinping" and "Barack
 Obama" are represented as blue circles.
- **Timestamp Nodes (Green Triangles)**: The period of François
 Hollande's presidency is represented as a green triangle, indicating
 the specific timeframe relevant to the query.

**2. Entity and Time Set Projections (Arrows)**:

- **Blue Arrows**: These represent entity set projections. For instance,
 an arrow from the entity node "Xi Jinping" to a logical operation
 indicates that the action of visiting countries is being queried
 specifically for Xi Jinping.
- **Green Arrows**: These represent time set projections. They connect
 the timestamp node of Hollande's presidency to other operations,
 applying the time filter to the activities of the entities involved.

**3. Logical Operations (Red Rectangles)**:

- **Visit Operations**: Separate logical operations check which
 countries each leader visited during Hollande's presidency. These
 operations are linked from the entity nodes (Xi Jinping, Barack Obama)
 through blue arrows, indicating entity-focused actions within the
 specified time, linked through green arrows.
- **AND NOT Operation**: This logical operator compares the outputs from
 the Xi and Obama visit operations. It filters out any overlaps, thus
 excluding countries visited by Obama from Xi's list.

**4. Final Output (Node)**:

- The output node at the end of the computation graph provides the
 answer to the query. It lists countries visited by Xi Jinping but not
 by Barack Obama during the defined timeframe. This is achieved after
 processing through the logical AND and NOT operation, which ensures
 only Xi's unique visits are counted.

<figure>
<img src="https://cdn-images-1.medium.com/max/940/0*xEF6vn1R_85HA6xH" />
<figcaption>Figure 3: The computation of time part in temporal operators
Before and After. (Lin et al., 2022) Article link: <a
href="https://arxiv.org/abs/2205.14307">https://arxiv.org/abs/2205.14307</a></figcaption>
</figure>

The computation graph in Figure 3 represents how these time adjustments
are made within the embeddings. It shows that only the time components
are modified in the highlighted colors leaving the entity-related parts
unchanged in faded gray color. It focuses solely on the temporal shift
implied by the query which is important for reasoning over knowledge
that changes over time.

### **Model Prediction in temporal context before and after the event**

<figure>
<img src="https://cdn-images-1.medium.com/max/940/0*gCoU-bQvkz2ubPny" />
<figcaption>Figure 4: Score distributions of Pt,
bPt and aPt, where s is
Citizen (India), r is Criticize or
denounce, o is Sadhu (India),
ground-truth answer of Pt is
2014–03–27. (Lin et al., 2022) Article link: <a
href="https://arxiv.org/abs/2205.14307">https://arxiv.org/abs/2205.14307</a></figcaption>
</figure>

- Pt is projection in time that shows the most likely time for the event
 to have occurred, which is aligned with ground truth date.
- bPt (Before Projection Time): bPt indicates time earlier than the peak
 for "Pt," as highlighted by the 'before' region in the graph.
- aPt(After Projection Time): aPt indicates time earlier than the peak
 for "Pt," as highlighted by the 'after' region in the graph.

<figure>
<img src="https://cdn-images-1.medium.com/max/940/0*UZSHCPLELDoiXMIs" />
<figcaption>Figure 5: Score distributions of Pt,
bPt and aPt, where s is
Citizen (India), r is Use
unconventional violence, o is Chief Secretary
Chandra, ground-truth answer of Pt is
2014–05–11. (Lin et al., 2022) Article link: <a
href="https://arxiv.org/abs/2205.14307">https://arxiv.org/abs/2205.14307</a></figcaption>
</figure>

Figure 4 & 5 uses the x-axis to detail the model's performance across
different types of queries, demonstrating how predictions align with
ground truth data over time. In simple words, the purpose of the above
visualization is to show the model's ability to predict when events
occur relative to a known point in time. ***"Pt"*** is the model's
direct prediction, while ***"bPt" and "aPt"*** show the model's
understanding of the temporal context before and after the event,
respectively.

### **Result**

Take this query example: Philippines denounced or criticized who on
2014--04--01?

<figure>
<img
src="https://cdn-images-1.medium.com/max/1004/0*Ja6buG4h-uwSHuA0" />
<figcaption>Table 1: Intermediate variable assignments and ranks for
example query. Correctness indicates whether the answer belongs to the
ground-truth set of answers.</figcaption>
</figure>

Taking another query example: On 2014--04--04, who consulted the man who
was appealed to or requested by the Head of Government (Latvia) on
2014--08--01?

<figure>
<img src="https://cdn-images-1.medium.com/max/977/0*SCTi1GRsq-phc64R" />
<figcaption>Table 2: Intermediate variable assignments and ranks for
example second query. Correctness indicates whether the answer belongs
to the ground-truth set of answers.</figcaption>
</figure>

In Table 2, the query "On 2014--04--04, who consulted the man who was
appealed to or requested by the Head of Government (Latvia) on
2014--08--01?" was taken and then TFLEX was used to execute a query and
get an answer. The model successfully classified the level of
difficulties of the answer type into Easy and Hard. It is also able to
classify which answer is correct and wrong. The table shows that TFLEX
gives simple answers "François Hollande", "Taavi Rõivas", and hard
answers "Andris Berzins" a higher ranking than incorrect answers "Angela
Merkel" and "Head of Government (Latvia)". This demonstrates how TFLEX
correctly determines the hard answer through sophisticated reasoning and
separates the correct responses from incorrect ones.

### **Industry Applications**

Temporal knowledge graphs (TKGs) have diverse applications:

1. **Recommendation Systems**: TKGs are used to model recommendation
 systems by analyzing users' past behaviours such as liking, buying,
 watching, reviewing, and so on. It can also offer more personalized
 and timely recommendations, such as suggesting products based on
 recent browsing and purchasing trends.
2. **Financial Application** In financial markets, TKGs can help in
 holding behaviours, trading behaviours, and financial events that
 aid in better investment decisions.
3. **Transportation**: In the field of transportation, temporal
 knowledge graphs can be used to improve traffic prediction models
 and enable intelligent transportation systems. (Yuan & Li, 2021)

### **Conclusion**

This post looks at how artificial intelligence is getting better by
using Temporal Knowledge Graphs (TKGs) and a special method called
Temporal Feature-Logic Embedding (TFLEX). TKGs are advanced versions of
knowledge graphs that include time information, which helps in
understanding and predicting complex patterns. TFLEX is designed to deal
with these time-related challenges and improves how we can answer
detailed questions about past and future events. This is useful in areas
like finance, transportation, and personalized recommendations, where
understanding time can make a big difference.

### **References**

- Lin, X., Xu, C., E, H., Su, F., Zhou, G., Hu, T., Li, N., Sun, M., &
 Luo, H. (2022, May 28). TFLEX: Temporal Feature-Logic Embedding
 Framework for Complex Reasoning over Temporal Knowledge Graph.
 arXiv.org. <https://doi.org/10.48550/arXiv.2205.14307>
- Li, M., Tong, P., Li, M., Jin, Z., Huang, J., & Hua, X. S. (2021, May
 18). Traffic Flow Prediction with Vehicle Trajectories. Proceedings of
 the . . . AAAI Conference on Artificial Intelligence.
 <https://doi.org/10.1609/aaai.v35i1.16104>
- Yuan, H., & Li, G. (2021, January 23). A Survey of Traffic Prediction:
 from Spatio-Temporal Data to Intelligent Transportation. Data Science
 and Engineering. <https://doi.org/10.1007/s41019-020-00151-z>

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0da8483262f9){width="1"
height="1"}

Latest findings in multiple research directions for tackling reasoning
and common sense challenges

<figure>
<img src="https://cdn-images-1.medium.com/max/894/0*2_PaHcp10KVBvRZq" />
<figcaption>Source: Generated using DeviantArt’s DreamUp</figcaption>
</figure>

### Author:

- Xuzeng He (**ORCID:**
 [0009--0005--7317--7426](https://orcid.org/0009-0005-7317-7426))

Knowledge Graphs, such as Wikidata, contain rich relational information
between entities and have been widely used as a structured format for
storing and representing relational information. With the surge of Large
Language Models (LLMs) nowadays (e.g. ChatGPT), there is a rising trend
in the research of combining Knowledge Graphs with Large Language Models
in multiple ways to solve various kinds of problems.

In this post, we introduce some of the latest findings where the use of
Knowledge Graph is combined with Large Language Models.

### Chain of History (CoH) reasoning

Temporal Knowledge Graph (**TKG**) is a subset of Knowledge Graphs
having some directions and timestamps. This allows capturing how
relationships evolve over time or the order in which they occur. It
holds significant practical value across various applications. For
example, a TKG might show that "Albert Einstein" (entity) "published"
(relationship with time) the theory of relativity in "1905" (timestamp).
Most advanced research on TKGs mainly focuses on predicting future facts
based on given historical facts.

Recent supervised methods primarily rely on Graph Neural Networks
(**GNNs**) to capture structural information (e.g. entities and
relationships). They can learn representations for entities and
relationships by aggregating information from neighboring nodes in the
graph. This helps identify patterns and connections within the TKG.
While GNNs capture structure, they might not fully grasp the semantic
meaning of entities, relationships, and timestamps in a TKG. They focus
on learning patterns from labeled data, which might not translate well
to understanding the underlying meaning of the data. Fortunately, with
the assistance of LLMs, their capability in this direction has been
gradually explored. However, issues still remain. For example,
over-complicated historical information may confuse LLMs and make it
hard for them to give accurate answers. When you input a large paragraph
to LLMs, models may struggle to select useful information and could
possibly generate an answer irrelevant to the question.

To address the existing issues, Chain-of-History (**CoH**) reasoning
method for TKG prediction is proposed. Instead of providing LLMs with
all histories at once in the traditional approach, CoH provides LLMs
with high-order histories (e.g. information not directly related to the
query) step-by-step. Specifically, CoH adopts LLMs to explore important
high-order history chains step-by-step and reason the answers to the
query only based on inferred history chains in the last step.

**First-Order Histories**

- These are the **individual events** or pieces of information
 considered most relevant to answering the query.
- The LLM analyzes the provided information and identifies these key
 elements.

**Second-Order Histories**

- These are formed by **combining** the first-order histories **in a
 specific order**.
- They represent **sequences of events** that could have led to the
 current situation or answer the query. Second-order histories go
 beyond isolated events and capture the **causal relationships**
 between them.

This process continues where chains of first-order histories, forming
more intricate event sequences are formed for third-order, fourth-order
etc histories.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*ae8Y-1ocdu-eSB95" />
<figcaption>An illustration of a two-step CoH reasoning procedure. In
the first step, LLMs are provided with only first-order histories and
asked to infer the most important histories. In the second step, LLMs
are provided with second-order history chains based on the inferred
first-order histories and asked to infer possible answers to the given
query. Then the answers inferred by LLMs and graph-based models are
adaptively fused to make the final prediction. Note that this only
serves as a two-step reasoning example, more steps can be executed with
CoH. (Yuwei Xia et al., 2024) Article Link: <a
href="https://arxiv.org/abs/2402.14382">https://arxiv.org/abs/2402.14382</a></figcaption>
</figure>

Extensive experimental results demonstrate the superiority of CoH, and
its effectiveness in enhancing the performance of graph-based models for
TKG prediction.

### Right for Right Response

Knowledge Graph Question Answering (**KGQA**) methods seek to answer
Natural Language questions using the relational information stored in
Knowledge Graphs (KGs). With the recent advancements of Large Language
Models (LLMs) and their remarkable reasoning abilities, there is a
growing trend to leverage them for KGQA. However, existing methodologies
have only focused on answering factual questions, such as "In which city
was Simone Biles born". In this case, questions involving commonsense
reasoning that real-world users may pose more often, such as "Do I need
separate visas to see the 'Venus of Willendorf' and attend the Olympics
this summer", are left out.

A verifiable methodology for performing KGQA using LLMs called ***Right
for Right Reasons* (R3)** is proposed to solve this issue. R3 casts the
problem of common sense KGQA as a tree-structured search, in which every
reasoning step is either grounded on KG facts, or based on surfaced
commonsense axioms, a key property that makes the reasoning procedure
completely verifiable.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*0NK-rUxHLtFPlTjR" />
<figcaption>An example workflow of common sense KGQA procedure using R3.
(Armin Toroghi et al., 2024) Article Link: <a
href="https://arxiv.org/abs/2403.01390">https://arxiv.org/abs/2403.01390</a></figcaption>
</figure>

In this example workflow, R3 first identifies the anchor entities(The
central entities mentioned in the query that are most relevant to
finding the answer) of a query and obtains the relevant sub-graph for
these entities. Next, it surfaces a commonsense axiom from the LLM that
will guide the reasoning steps in that branch of the search tree. Then,
at each depth level of the tree, it checks whether the common- sense
axiom can be satisfied with the available KG facts, and if possible,
provides an answer grounded on a subset of them. If the available KG
triples are insufficient, by backward-chaining from the axiom, it
selects the next entity to obtain its relevant KG sub-graph to continue
the search. Each branch can continue up to a maximum depth, and if an
answer is not obtained at its bottom, a new commonsense axiom will be
surfaced which will guide search in a new branch until the search tree
reaches its maximum breadth.

Experimental evaluations across three different tasks --- question
answering, claim verification, and preference matching --- showcase R3
as a superior approach, outperforming existing methodologies and notably
reducing instances of hallucination and reasoning errors.

### KnowPhish

Phishing attacks are one of the most impactful types of scams, harming
both individuals and businesses. Many efforts have been made to counter
phishing attacks. Among these, Reference-Based Phishing Detectors
(**RBPDs**), which work by comparing the logos on a target webpage to a
known set of logos, have been established as the state-of-the-art
phishing detection paradigm, garnering considerable research attention.

Specifically, an RBPD consists of a Brand Knowledge Base (**BKB**)
containing brand information (e.g., the logos and legitimate domains of
brands) and a detector backbone that uses information from this BKB for
phishing detection. To detect if a web page is a phishing page, RBPDs
first identify the webpage's brand intention, that is, the brand that
the webpage presents itself as (e.g., a webpage with a Adobe logo and
appearance has the brand intention of Adobe). Then, if the webpage is
detected to have an intention of a certain brand, but its domain does
not match the legitimate domains of that brand, the webpage is
classified as a phishing attempt.

However, RBPD has its limitations as well, such as not being able to
identify textual Brand Intention. Therefore, a large-scale multimodal
BKB named KnowPhish containing rich logo variants, aliases, and domain
information of each brand was recently constructed and publicly
released, along with a Large Language Model (LLM)-based approach to
identify the brand intention of webpages in conjunction with the alias
information in KnowPhish. This approach can be directly integrated with
any standard visual RBPD, enabling it to detect brand intention through
both visual and textual modalities. The resulting multimodal phishing
detection approach, named KnowPhish Detector (**KPD**), is able to
detect phishing webpages with or without logos.

KnowPhish has been shown to dramatically improve the performance of
RBPDs. It is also significantly faster than DynaPhish, another detection
framework by 30 or more times, especially when using image-based RBPDs.
Moreover, incorporating the multimodal approach(incorporating various
data sources such as images, text, audio, etc) KPD, can substantially
boost the number of detected phishing webpages.

### Conclusion

This post explored how Knowledge Graphs (KGs) and Large Language Models
(LLMs) are being combined to address challenges in various tasks. The
key takeaways are:

- **Temporal Knowledge Graphs (TKGs) with LLMs:** The Chain-of-History
 (CoH) reasoning method offers a promising approach for TKG prediction.
 By breaking down complex historical information into steps for LLMs,
 CoH improves reasoning accuracy, especially when combined with
 graph-based models.
- **KG Question Answering (KGQA) with LLMs:** The Right for Right
 Reasons (R3) methodology tackles common-sense KGQA. R3 leverages LLMs
 to surface commonsense knowledge and guide the reasoning process. This
 verifiable approach reduces errors and hallucinations compared to
 existing methods.
- **Phishing Detection with LLMs:** The KnowPhish approach demonstrates
 the power of LLMs in enhancing Reference-Based Phishing Detectors
 (RBPDs). By incorporating a multimodal Brand Knowledge Base (BKB) and
 LLM-based brand intention identification, KnowPhish significantly
 improves phishing detection rates and speeds compared to traditional
 methods.

Overall, the combination of KGs and LLMs opens exciting possibilities
for tackling complex tasks that require reasoning and knowledge
representation. As research in this area progresses, we can expect even
more innovative applications that leverage the strengths of both these
technologies.

### References

- Xia, Y., Wang, D., Liu, Q., Wang, L., Wu, S., & Zhang, X. (2024).
 Enhancing Temporal Knowledge Graph Forecasting with Large Language
 Models via Chain-of-History Reasoning.
- Toroghi, A., Guo, W., Pour, M. M. A., & Sanner, S. (2024). Right for
 Right Reasons: Large Language Models for Verifiable Commonsense
 Knowledge Graph Question Answering.
- Li, Y., Huang, C., Deng, S., Lock, M. L., Cao, T., Oo, N., Hooi, B., &
 Lim, H. W. (2024). KnowPhish: Large Language Models Meet Multimodal
 Knowledge Graphs for Enhancing Reference-Based Phishing Detection.

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=660588951c16){width="1"
height="1"}