Carp posted:

GPT does not regurgitate text. After training, the neural net's parameters are finalized, and it does not have access to any form of text. Base GPT does not reference a database. However, some products built on GPT give it access to various data sources, such as Bing Chat with live search results or Wolfram Alpha (GPT-4 Plugin). How a data source is used during a session with a GPT product is determined by the natural language instructions given to GPT before the session starts and the instructions given by the user during their session. Nearly all the abilities that people are excited about are emergent and were discovered in, or coaxed from, GPT's neural net, which is a store of relationships, both large and small in scope, using tokens itself discovered while training. No human ever gave it an explicit description of language. In fact, it trained on numerous different languages all at once, consuming a huge collection of free-form, unlabeled text using self-supervised learning. It was not given labeled sets of data.