When the Open AI neural network GPT-3 was introduced in May 2020, it set a new standard in deep learning and was considered the most advanced at the time. The AI model could generate text that was virtually indistinguishable from what was written by a human. But just 10 months later, researchers from the Beijing Academy of Artificial Intelligence announced creating their own generative neural network model called Wu Dao, capable of doing everything that GPT-3 can do and even more.
Just three months later, Wu Dao 2.0 appeared with 1.75 trillion parameters, which is 10 times more powerful than GPT-3 and 150 billion more parameters than Google Switch Transformers. Chinese experts first developed an open-source learning system similar to Google Mixture of Experts called FastMoE. It allowed training a neural network model both on supercomputer clusters and on conventional GPUs. This gave the system a lot of flexibility as it did not require proprietary hardware like Google did and could run on standard hardware.
With all this computing power, the new neural network has a huge set of capabilities. Unlike most deep learning models, which often perform a single task, Wu Dao is multimodal. That is, it can perform multiple tasks. In theory, it is similar to the AI that Facebook uses to combat hate and misinformation.
Researchers have demonstrated Wu Dao’s ability to perform tasks in natural language processing, image and text generation, and image recognition. The neural network can write essays, poems, and couplets in traditional Chinese. Still, it can also create alternative text based on a static image and generate almost photorealistic images from the description. Wu Dao mimics speech, creates recipes, and predicts the three-dimensional structure of proteins, similar to AlphaFold.
Almost 5 TB of data was used to train Wu Dao 2.0. Several dozen companies have already become interested in the development.