When GPT-4 was released in March this year, it was branded as an advanced model with multimodal capabilities. However, multimodality was nowhere in sight. After almost six months, OpenAI released a string of updates last week, with the most notable one being the image and voice feature, making GPT-4 truly multimodal, and finally bringing in the ‘Vision’ feature.
Table of contents
As showcased by OpenAI’s co-founder Greg Brockman in the demo video for explaining GPT-4 functionalities earlier this year, the varied uses of GPT-4 V(ision) have been put to test and the results have been incredible. Here are some of the amazing features of GPT-4V.
Identifying Objects
Be it a plant, animal, character or any random object, GPT-4 has been able to correctly identify it from an image. Furthermore, it is able to generate descriptive details about the object. In the below screenshots, ChatGPT has been able to rightly identify the main plant without any descriptive input prompt, and the character ‘Waldo’, respectively (below).
Transcribing Text
If we input an image with any form of text into ChatGPT Plus, the model is able to transcribe the content from the image. In the screenshot below, the image contains medieval writing from philosopher and writer Robert Boyle’s manuscript.
Deciphering Data
The model is able to easily read graphs and charts and in all formats, and is able to draw conclusions based on them. In the below screenshot, a bar graph of performance of two models on various competitive exams are shown.
Processing Multiple Conditions
The model can also comprehend and process images with multiple conditions. For example, in the image below, it has read a set of instructions to arrive at an answer.
Teaching Assistant
Acting like a virtual teacher, the chatbot can converse with a user to understand topics from various subjects. In the below tweet, a diagram has been elaborately explained as per given instructions.
Upgraded Coding
With ChatGPT Code Interpreter already out there, GPT-4 V(ision) pushes its coding capabilities to another level. By simply uploading an image, you can perform a wide variety of coding-related functions.
In the below post, a user has been able to convert an image to a live website.
Enhanced Design Understanding
With a probable flair for design, the chatbot is able to identify various architectural designs. It is also able to suggest design changes based on custom instructions provided by a user.
