A phone displaying open Ai sat on top of a laptop with ChatGPT open

GPT 4 Image Input: How to use ChatGPT Image Input feature

Can ChatGPT read images?

Silent PC Review is reader-supported. When you buy through links on our site, we may earn an affiliate commission. Learn More

Table of Contents

In March 2023, OpenAI launched the latest version of its premium multimodal language model, GPT 4, allowing image analysis alongside traditional text generation. The ChatGPT image input function means that the AI can identify elements within an image that you upload, and then produce text based on it to whatever prompt you like. The possible uses of such a model are extensive, with potential impacts on a wide range of fields including entertainment, education, and commerce. With this new advancement, you can use code, instructions, and images to build incredible things on the GPT 4 API.

✓ TLDR

Which plans can currently use image inputs?

Initially, you could only access the API as a developer, however, it was later introduced for ChatGPT Plus subscribers and ChatGPT Enterprise. If you are using the free version of ChatGPT you will not be able to use this feature.

GPT 4 Image Input: How to Upload Images to ChatGPT

If you are a ChatGPT Plus or Enterprise subscriber, you already have access to this GPT 4 image input feature when you log in to get started upload a photo, and start chatting. There doesn’t appear to be any image upload limit but OpenAI states that if you start to encounter issues, consider reducing the image size or quantity.

There are a few ways to use GPT 4 to read images, but the first step is to upload them. On a smartphone, you can take a photo using the camera icon in the ChatGPT App and, if necessary, highlight elements of the image that you’d like the AI to focus on. In the conversation, you can ask ChatGPT about objects in your image or even ask it to analyze documents and other visual content.

Alternatively, you can use older images from your camera roll or picture library, uploaded as you would do with any other messenger platform. However, this is limited to standard, static images for now: video or GIF formats are not supported on GPT 4.

How Does GPT 4 Read Images?

While an AI language model does not inherently process visually like a computer vision model, it can still be used in conjunction. The GPT 4 model can process text and picture input, allowing for natural language, code, instructions, or artificial opinions to be received as a response. This means that ChatGPT describes an image by analyzing the data in the same way as it analyses a textual prompt. Once you upload an image, it seeks out patterns or known entities and formulates them into a response.

ChatGPT can read images but generally needs some form of prompt or instruction for any kind of meaningful response. It could be asked to identify a brand or an item in the frame, or describe the frame as a whole in plain text. This extends to graphics such as charts or graphs. This is incredibly useful for creating alternate text as image captions, or for identifying specific objects. Like the disclaimer says, however, ChatGPT can make mistakes, so do not take its word as gospel!

GPT 4 Limitations

The new GPT 4 model shares some of the same limitations as we saw with earlier ChatGPT models. It is still not 100% reliable and admits when making mistakes. OpenAI frequently states that:

Great care should be taken when using language model outputs, particularly in high-stakes contexts, with the exact protocol (such as human review, grounding with additional context, or avoiding high-stakes uses altogether) matching the needs of a specific use-case.

OpenAI

Other limitations OpenAI makes its users aware of are:

  • The feature can struggle to interpret specialized medical images (such as CT scans) – so should not be used for medical advice.
  • The model can also struggle with “Non-Latin” alphabets on images.
  • The model can misinterpret rotated images/ text.
  • It may struggle with graphs and other visual elements.
  • It may struggle with precise spatial localization and can only give approximate counts for objects in images.
  • It can struggle to read panoramic or fisheye images

GPT 4 was trained like its previous models: to predict the next word in a document using publicly available online information and licensed data – meaning that the pitfalls still betray it. OpenAI draws from a nebulous array of sources to generate an answer, saying:

The data is a web-scale corpus of data including correct and incorrect solutions to math problems, weak and strong reasoning, self-contradictory and consistent statements, and representing a great variety of ideologies and ideas.

OpenAI

In other words, GPT 4 uses everything at its disposal to come up with an answer, with no real bias to the information source. It can get things wrong and can plagiarize or write unreliable text. We are still far from an entirely trustworthy ChatGPT model.

ChatGPT Image Input FAQs

Can ChatGPT Generate Images?

While not capable of processing image input directly, ChatGPT can describe images once uploaded. These descriptions can then be used as input for image-generating tools like DeepAI, DALL·E, and Midjourney.

Is ChatGPT free?

Whilst the full GPT models such as GPT 4 require a subscription to access, Open AI have a free-to-use basic model of ChatGPT available to all for free.

In Summary

On top of being an ever-evolving language AI, ChatGPT is still being worked on by developers at OpenAI. We’ve already seen the incredible image-processing abilities of AI, removing imperfections or adding anything to an image with Photoshop’s Generative Fill system. Though they have launched GPT 4 with picture input, there is no telling what additional image features could be added in the future. If you want to generate text prompts from images, GPT 4 can read images and provide concise descriptors or analysis. However, for turning text prompts into images, you’ll have to settle for other generative AIs such as Midjourney or DALL·E for now. Google Bard also looks to incorporate visual AI, so you are spoiled for choice.

About the Author