GPT inputs outgrow world chip and electricity capacity

by quirkot on 2/13/2024, 3:48 PM with 14 comments
  • by fulafel on 2/13/2024, 4:08 PM

    Badly editorialized title.

    Regarding the substance of the article, the curve from the 3 data points (1: 100x 2: 25x 3: 25x) could fit lots of different ways besides "growing 30x per generation".

  • by danielscrubs on 2/14/2024, 8:44 AM

    Really wish AMD could hurry up and implement a transparent module for gpu computing into llvm.

    It’s not good that GPUs are this opaque.

  • by dekhn on 2/13/2024, 11:47 PM

    The are all the same arguments that wrongly predicted that DNA sequencing would overtake hard drive storage capacity.

    People aren't going to do truly uneconomic things just to scale language models exponentially.

  • by marsissippi on 2/13/2024, 4:13 PM

    Interesting Pull quote:

    GPT-4 needed about 50 gigawatt-hours of energy to train. Using our scaling factor of 30x, we expect GPT-5 to need 1,500, GPT-6 to need 45,000, and GPT-7 to need 1.3 million.

  • by RecycledEle on 2/14/2024, 5:09 PM

    New technologies experience very rapid exponential growth for a while.

    This should not be a surprise.