Talk for USENIX'05, April 2005

At the moment, I have these html "slides" available at:
www.rohan.sdsu.edu/faculty/vinge/misc/u05

• Don't buy a trend extrapolation, however sophisticated
• Don't buy a single grand projection
• Work out a variety of credible scenarios
  • Scenario-based planning
  • For example the future of oil prices and the founders of GBN
• How to use good scenarios
  • Confess: this approach does not cover the space of all surprises! (But the hope is that it gives us greater flexibility than betting the farm on one vision of the future.)
  • Watching for symptoms
  • Taking advantage of opportunities
  • Maybe, preventing the future
• My strategy in this talk
  • Four scenarios, marking out some extremes of the probability space.
  • I base 3 of the 4 scenarios on the future of Moore's Law

www.rohan.sdsu.edu/faculty/vinge/misc/u05

We have enjoyed a long period of exponential growth in hardware performance. Without the hardware, we don't have practical software, but what effect can improving hardware have on software?

In real life, what sort of things can happen to exponential growth?

• Exponential growth
• Exponential growth with saturation
• Exponential growth with catastrophic collapse

I'll take each of these as a future for hardware trends, imagine how that development might come to pass and what the resulting software environment might be like.

How could this scenario happen?

• Through the mother of all software failures? Maybe. Certainly, enormous service disruptions are out there in our future -- caused by software and deliberate attacks on software. My intuition (my scenario, after all :-) is that as long as we have heterogeneous, multiple software solutions, that software will not be the cause of a complete failure.
• Semiconductor technology as the single failure point.
  • The homogeneity of certain semiconductor parts. (This threat is a matter of principle, not just the obvious and credible threat of EM Pulse attacks.)
  • The ubiquity of semiconductor parts
  • The trend that almost all of us in this room are pushing hard every day.
  • Most of us have a treacherous psychological blindspot: If a behavior runs a catastrophic but unquantifiable risk, but the behavior yields progressively more magnificent rewards, then we are happy to ignore the possibility of disaster. That is our current situation with computing, in particular with embedded microcontrollers (and networks of such devices).

The use of programmatic solutions implemented in embedded microcontrollers is a mind-bogglingly powerful economic miracle. It harnesses Moore's Law to make most classical goods as cheap as dirt. It opens the way to a standard of living that towers above the past. It also puts us way out on a limb. Is this a fact we can ignore? If not, what can we do about it?

No doubt, I'm going to continue be peddling the wondrous upside of computation in my fiction, but more fiction and scenarios about this risk should be undertaken, and perhaps suggestions for solutions. Some mitigating possibilities:

• We should encourage old-time (no semi-conductor) hobbyists. This won't save an absolutely large number of lives, but it might save civilization.
• Known threats to this single failure point should be much more intensively discussed (and without the foregone conclusion that since we really really need this tech, the threat must be minor).

• Dies the Fire, by S. M. Stirling, (ROC). (The background is very different from what is imagined in this talk, but the actual crash is similar and vividly described.)
• The End of Man, by Hannes Alfvén, (Coward-McAnn). (The crash is just a small part of the book, but vividly described, even though it happens at a distance.)
• Others?

In terms of hardware trends, this one comes with exponential growth with saturation

For at least ten years, people have been saying that Moore's Law is soon to run out of steam, so there should be lot's of people who buy into this scenario.

• Maybe the hardware people will run out of inventiveness.
• Maybe the software people find that they can't manage complexity above a certain limit.

It would be interesting to observe this leveling off. Some possible symptoms (depending on the underlying cause from above):

• More and more software bloat with less and less payoff
• Edelson's law (due to Steve Edelson, but possibly muddled here): If knowledge is increasing exponentially, then "The number of inventions that could be made -- but aren't -- increases exponentially."
• Enormous software project failures
• Murphy's counterpoint to Moore's Law "The maximum possible effectiveness of a software system increases in direct proportion to the log of the effectiveness (ie, speed, bandwidth, memory capacity) of the underlying hardware."
• Ultimately, the above problems would mean physics and materials science research would flounder -- and we'd see a leveling out of the hardware trends.
• Very likely, some types of hardware progress would continue forward, even as others leveled off.
  • Simple, large structures (RAM and DSP logic)
  • Related consumer electronics

After equilibrium was achieved, some (naive?) programmers might give a deep sigh of relief: "Finally, we'll have time to do things right!"

Even more than now, programming would come down to a choice between software archeology and re-inventing the wheel.

• A Deepness in the Sky, by Vernor Vinge (Tor Books). This story takes place thousands of years after hardware/software equilibrium is achieved.

In terms of hardware trends, this one comes from exponential growth continuing, at least long enough to create our post-human successors.

This is the scenario I've been peddling for a long time:

• From 1983: "First Word" OpEd piece in Omni magazine, January 1983, page 10.
• From 1993: "The Technological Singularity", NASA Lewis VISION-21 Symposium, 30-31 March 1993. http://www.rohan.sdsu.edu/faculty/vinge/misc/singularity.html

It's a little extreme, but in the wider view of future scenarios, I think it's the most likely of the non-disasters. (For numerous deadly disasters, see Martin Rees, Our Final Hour (nonfiction) (Basic Books).)

• By analogy with the use of "singularity" in Math
  • Not necessarily a place where anything becomes infinite
    • A qualitatively different step of progress than in our technological past
  • A place where the rules profoundly change
• By analogy with the use of "singularity" in Physics
  • What comes after is intrinsically less knowable/predictable than previous technological advances.
  • The apocalyptic endpoint of radical optimism

The Singularity as the dual of Fermi's Paradox

• What if: AI (Artificial intelligence) research succeeds?
• What if: The internet itself attains unquestioned life and intelligence?
  • Gregory Stock, Metaman (nonfiction)
• What if: IA (Intelligence Amplification) occurs as the radical endpoint of Human/computer interface research?
  • Poul Anderson, "Kings Who Die"
  • Vernor Vinge, "Bookworm, Run!"
• What if: Fine-grained distributed systems are aggressively successful?
  • Communicating, embedded micros
  • The physical world awakes

• The end of software engineering. More precisely, the partition of software projects into complexity categories:
  • Programs that can be mathematically proof-verified. Wherever this can be done, it should be; however, my guess is the domain for such certainty will remain with extremely-constrained applications.
  • Projects that are susceptible to engineering approaches. There should still be many projects like this, ones where quantitative analysis and a step-by-step implementation of an overall vision produces effective results. These are projects in which we can at least fool ourselves that there is a well-defined product specification.
  • Mongo Projects. These are things people really really want, but the desired product is usually very large, involving vague or incomplete specification (even though the customers are sure they will recognize success when they see it).
• An organizing principle for Very Large Data Systems:
  • Reality should be its own database. "Every object knows what it is, where it is, and can talk to nearby peers (and thus in principle talk across the world)." (Cf the term "thalience" in Ventus by Karl Schroeder (Tor Books).)
  • And so cyberspace leaks into the real-world. There is a heuristic for invention in the new age: In the 20th century, we often used real world metaphors in our programming. Now turn that around: almost every gimmick/insight that we've been using in computation will have some real world analog (Google on 'Accenture "virtual collaborating double"').
• The rise of biological models for problem solving. People have been working on this for a long time ("genetic programming", "neural nets", ...). Now we are entering an era where the hardware may be up to the promise.
  • Precedent: Consider the surprises that numerical analysts (for example, the differential equations people) suffered when automatic computation arrived.
  • Programming as an exercise in evolution
  • Programming as an exercise in embryology
  • Programming as an exercise in training/teaching
  • Programming as an exercise in sociology
    • "Maneki Neko", by Bruce Sterling, F&SF, May 1998
  • Software with levels of "resource abuse" that make the most wasteful 2005 xml/java programmers look like Scrooge. Worse, the exact source of the waste will not be as obvious as in our time.
  • What about trusting life-critical applications to such "software"?

• Nightmare analogies -- our treatment of animals
  • Vernor Vinge, "The Cookie Monster" currently at http://www.analogsf.com/0501/cookiemonster.shtml It is very likely that plenty of unpleasant jobs will remain for critters of about human intelligence.
• Hard takeoff: transition into Singularity takes only a few days (or a few hours!). This version sounds tailor made for nightmares.
• Soft takeoff: transition into Singularity takes a years. Imagine the distributed scenario discussed above. There might be time to plan, time to create good feeling with the new entities
• Other analogies http://www-rohan.sdsu.edu/faculty/vinge/misc/evolution.html

In my 1993 "Technological Singularity" essay
http://www.rohan.sdsu.edu/faculty/vinge/misc/singularity.html
I have a pretty good bibliography of related fiction (and non-fiction). This is slightly elaborated upon at
http://www.rohan.sdsu.edu/faculty/vinge/misc/WER2.html

As of 2005, there are lots of stories in this category. Many people writing hard science fiction feel a necessity for addressing this possibility -- if only to explain why they can keep it at least partially offstage! About half of modern interstellar adventure has a background that involves post-human or transhuman players (including almost all my stuff).

In the last few years, there have been more stories that look at the near future (when, after all, the Singularity is likely to happen) and try to do the nearly impossible -- that is, write a fun story with characters who are slipping beyond human ken. Charles Stross's Accelerando, to appear (Ace Books) is one of the most interesting of this genre.

• Vernor Vinge, presentation at Computers, Freedom, and Privacy conference, 1996. Here is a redrawing of my main graphic.
• John Walker, "The Digital Imprimatur" http://www.fourmilab.ch/documents/digital-imprimatur

Examples solicited. I don't have any clearcut fiction examples of this exact idea. I waved my hands at ULE in A Deepness in the Sky (Tor Books). Also, Ubiquitous Law Enforcement is a major background element in the novel I am currently finishing up (working title: Rainbows End) -- where I use ULE to make plausible the success of the secret police against the hackers. :-)