Large Language Models (LLMs) have gathered an enormous amount of attention and recognition among the many Artificial Intelligence (AI) community in recent months. These models have demonstrated great capabilities in tasks including text summarization, query answering, code completion, content generation, etc.
LLMs are steadily trained on inadequate web-scraped data. Most of the time, this data is loud, unstructured, and never necessarily expressed clearly. Following the present scaling principles, which indicate that as the scale of the model increases, computational power and data quantity also needs to increase proportionately, comes as a challenge.
There are two important limitations. Firstly, there may be the numerous computational cost and time involved in pre-training. Secondly, there may be the approaching problem of the scarcity of high-quality data available on the Internet. In recent research, a team of researchers from Apple and Carnegie Mellon University has addressed these issues by introducing the concept of Web Rephrase Augmented Pre-training (WRAP).
WRAP is an revolutionary method that makes use of an already-existing, instruction-tuned LLM. This LLM is used to paraphrase websites into particular styles, including mimicking the tone of Wikipedia or converting text into an answer-question format. The important goal of WRAP is to enhance LLMs’ pre-training by adding each real and artificially rephrased data.
The primary features of WRAP are as follows:
- Pre-training Efficiency: Applying WRAP to the noisy C4 dataset considerably accelerates pre-training, around 3 times faster. This effectiveness is critical in reducing the high expenses and time commitment often related to LLM training.
- Enhancement of Model Performance: WRAP makes the model perform higher when run throughout the same computational budget. Using different subsets of the Pile, a large-scale dataset used for training and assessing LLMs reduces ambiguity by greater than 10%. It improves zero-shot question-answer accuracy by over 2% for 13 different activities.
- Rephrasing Web Documents: WRAP uses a medium-sized LLM to paraphrase documents from the online into several styles. This method is different from creating latest data since it improves already-existing content while preserving the unique information’s quality and variety.
There are two important advantages to the synthetic data produced by WRAP. Firstly, it includes a spread of styles that reflect the range of languages utilized in applications farther down the road. With this diversity, the LLM is healthier prepared for a greater variety of real-world events. Secondly, the synthetic data rephrased is of a better quality than the raw web-scraped data. This quality enhancement results from language that’s more ordered and cohesive, as this promotes more efficient model learning.
In conclusion, WRAP is a giant advancement in the sphere of LLM pre-training. Through the usage of superior-quality, different-style synthetic data, WRAP not only expedites the training process but additionally improves the general performance of LLMs. Given the abundance of low-quality web data and the resource-intensive nature of classic LLM training approaches, this approach presents a possible way forward.
Check out the Paper. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter and Google News. Join our 36k+ ML SubReddit, 41k+ Facebook Community, Discord Channel, and LinkedIn Group.
Don’t Forget to hitch our Telegram Channel
This article was originally published at www.marktechpost.com
