What is GPT-3?
GPT-3 stands for Generative Pre-trained Transformer 3, and it is the third version of the language model that Open AI released in May 2020. It is generative, as GPT-3 can generate long sentences of unique text as the output. Pre-trained means that the language model has not been built with any special domain knowledge, but it can complete domain-specific tasks like translation, summarizing texts, and a lot more which indeed makes it The most innovative language model that has ever existed.
It works by predicting the next token given a sequence of tokens and can do so for NLP tasks it hasn’t been trained on. After seeing just a few examples, it reached state-of-the-art levels in some benchmarks such as machine translation, Q&A, and cloze tasks.
GPT-3 was trained with huge Internet text datasets — 570GB in total. When it was released, it was the largest neural network with 175 billion parameters (100x GPT-2)
What GPT-3 can do?
Here’s a list of what GPT-3 can do, with links :
Nonfiction: Dialogue, impersonation, essays, news articles, plot summaries, tweets, teaching.
Professional: Ads, emails, copywriting, CV generation, team management, content marketing, note-taking.
Code: Python, SQL, JSX, React app, Figma, javascript, CSS, HTML, LaTeX
Creativity: Fiction, poetry, songs, humor, online games, board games, memes, cooking recipes, guitar tabs, write in your unique style.
Rational skills: Logic, uncertainty, common sense, analogies, concept blending, counting, anagrams, forecasting.
Philosophy: Meaning of life, number 42, responses to philosophers.
Examples:
Generating “Interview Questions”
Example [Input]: “Create a list of questions for my interview with my prospective girlfriend
Example [Generated Output]
- What are your favorite hobbies?
- What are your favorite TV shows?
- What are your favorite books?
- What are your favorite foods?
- What are your favorite places to visit?
- What are your favorite sports teams?
- What are your favorite movies?
Summarizing from A Given Text (tl;dr:)
Example [Input]
“The field of Artificial Intelligence (AI), has been advancing rapidly, especially since the start of ImageNet competition for Computer Vision in the early 2010s. Convolutional Neural Networks (CNN) with deep learning in architecture in a supervised learning approach opened up the possibilities for many practical applications with very high prediction accuracy. The capabilities include image classification, image segmentation, object detection, and generative modeling; applied to areas such as face recognition, gaming, robotics, and many more.
One of the challenges in supervised learning has been the need for a huge amount of data (and data labeling) to train the model to achieve the best possible trained model. Although, this has been gradually shifting with the few-shots and zero-shot learning approach. Another challenge is the infrastructure requirement to train the large models, fine-tuning the hyperparameters to optimum ones. Not to mention doing the inferencing, and running the trained model.
For non-vision-related, Natural Language Processing (NLP) and Natural Language Understanding have been significantly improved. GPT-3 is one of the latest models from OpenAI. GPT-3 is even inspired to do unsupervised learning for images and videos, the ones that are usually approached with supervised learning.
Are we approaching Artificial General Intelligence (AGI)? Well, time will tell.
tl;dr:”
Example [Generated Output]
AI is getting better, especially for image recognition and NLP. The infrastructure is getting better, with cloud computing and GPUs. The data is getting better, with few-shots and zero-shot learning. The hardware is getting better, with GPUs and TP
Bottom line
GPT-3 is a glimpse of the bright future in NLP, helping to generate code, meaningful pieces of texts, translation, and doing well with different tasks. Also, it has its limitations and ethical issues like generating biased fragments of text. All in all, we are witnessing something interesting, as it always used to be in NLP.
Reference: Understanding GPT-3 In 5 Minutes