We usually decide that problems are hard because smart people have worked on them unsuccessfully for a long
time. It's easy to think that this is true about AI. However, the past five years of progress have shown that the
earliest and simplest ideas about AI - neural networks - were right all along, and we needed modem hardware
to get them working.
Historically, AI breakthroughs have consistently happen with models that take between 7-10 days to train. This
means that hardware defines the surface of potential AI breakthroughs. This is a statement about human
psychology more than about AI. If experiments take longer than this, it's hard to keep all the state in your head
and iterate and improve. If experiments are shorter, you'll just use a bigger model.
It's not so much that AI progress is a hardware game, any more than physics is a particle accelerator game. But if
our computers are too slow, no amount of cleverness will result in AGI, just like if a particle accelerator is too
small, we have no shot at figuring out how the universe works. Fast enough computers are a necessary
ingredient, and all past failures may have been caused by computers being too slow for AGL
Until very recently, there was no way to use many GPUs together to run faster experiments, so academia had the
same "effective compute" as industry. But earlier this year, Google used two orders of magnitude more compute
than is typical to optimize the architecture of a classifier, something that usually requires lots of researcher time.
And a few months ago, Facebook released a paper showing how to train a large ImageNet model with near-linear
speedup to 256 GPUs (given a specially-configured cluster with high-bandwidth interconnects).
Over the past year, Google Brain produced impressive results because they have an order of magnitude or two
more GPUs than anyone. We estimate that Brain has around 100k GPUs, FAIR has around 15-20k, and
DeepMind allocates 50 per researcher no questions asked, and rented 5k GPUs from Brain for AlphaGo.
Apparently when people run neural architecture search at Brain, it eats up everyone's quotas at DeepMind.
We're still missing several key ideas necessary for building AGL How can we use a system's understanding of
"thing A" to learn "thing B" (e.g. can I teach a system to count, then to multiply, then to solve word problems)?
How do we build curious systems? How do we train a system to discover the deep underlying causes of all types
of phenomena - to act as a scientist? How can we build a system that adapts to new situations on which it hasn't
been trained on precisely (e.g. being asked to apply familiar concepts in an unfamiliar situation)? But given
enough hardware to run the relevant experiments in 7-10 days, history indicates that the right algorithms will be
found, just like physicists would quickly figure out how the universe works if only they had a big enough
particle accelerator.
There is good reason to believe that deep learning hardware will speed up 1Ox each year for the next four to five
years. The world is used to the comparatively leisurely pace of Moore's Law, and is not prepared for the drastic
changes in capability this hardware acceleration will bring. This speedup will happen not because of smaller
transistors or faster clock cycles; it will happen because like the brain, neural networks are intrinsically
parallelizeable, and new highly parallel hardware is being built to exploit this.
Within the next three years, robotics should be completely solved, AI should solve a long-standing unproven
theorem, programming competitions should be won consistently by Ais, and there should be convincing chatbots
(though no one should pass the Turing test). In as little as four years, each overnight experiment will feasibly use
so much compute compute that there's an actual chance of waking up to AGI, given the right algorithm- and
figuring out the algorithm will actually happen within 2-4 further years of experimenting with this compute in a
competitive multiagent simulation.
To be in the business of building safe AGI, OpenAI needs to:
1. Have the best AI results each year. In particular, as hardware gets exponentially better, we'll have
dramatically better results. Our DOTA and Rubik's cube projects will be impressive results for the current
level of compute. Next year's projects will be even more extreme, and what's realistic depends primarily
on what compute we can access.
2. Increase our GPU cluster from 600 GPUs to 5000 GPUs ASAP. As an upper bound, this will require a
capex of $12M and an opex of $5-6M over the next year. Each year, we'll need to exponentially increase
our hardware spend, but we have reason to believe AGI can ultimately be built with less than $1 OB in
hardware.
3. Increase our headcount: from 55 (July 2017) to 80 (January 2018) to 120 (January 2019) to 200 (January
2020). We've learned how to organize our current team, and we're now bottlenecked by number of smart
people trying out ideas.
4. Lock down an overwhelming hardware advantage. The 4-chip card that Jim Keller says he can build in 2
years is effectively TPU 3.0 and (given enough quantity) would allow us to be on an almost equal footing
with Google on compute. The Cerebras design is far ahead of both of these, and if they're real then having
exclusive access to them would put us far ahead of the competition. We have a structural idea for how to
do this given more due diligence, best to discuss on a call.
2/3/4 all will ultimately require large amounts of capital. If we can secure the funding, we have a real chance at
setting the initial conditions under which AGI is born. Increased funding needs will come lockstep with
increased magnitude of results. We should discuss options to obtain the relevant funding, as that's the biggest
piece that's outside of our direct control.
Progress this week:
• We've beat our top lvl test player (he's top 30 in North America at lvl, and beats the top lvl player about
30% of the time), but the bot can also be exploited by playing weirdly. We're working on understanding
these exploits and cracking down on them.
o Repeated from Saturday, here's the first match where we beat our top test player:
ht:tP-s ://www.Y-Outube.com/watch?v= FBoUHaY-7XB I&feature=Y-outu. be&t= 34 5
o Every additional day of training makes the bot stronger and harder to exploit.
• Robot getting closer to solving Rubik's cube.
o The improved cube simulation teleoperated by a human: https://drive.google.com/open?
id=0B60rCY-4P2FOIQ315UTdY-WmZ4VzQ
• Our defense against adversarial examples is starting to work on ImageNet
o We will completely solve the problem of adversarial examples by the end of August.
We also have four strong researchers leaving OpenAI:
• Peter Xi Chen, Rocky Yan Duan, and Jonathan Ho are leaving to start a robotics startup to solve all
robotics manufacturing in China. They're highly motivated to start a company (Peter and Rocky had an
unsuccessful startup before OpenAI).
• Alec Radford is leaving due to a mismatch in research interests (he doesn't believe AGI is near, and wants
to work on what he views as shorter-term problems).