Why Friendly AI research is critical: even if you give an AI nice sounding directives, it can be hard to know how far such an alien mind will take something. We take for granted all the other beliefs going around in our heads, such as that a hug shouldn’t be that strong, partly because we aren’t powerful enough to take things that far. The often discussed situation is that of just directing an AI to make people happy. What counts as a person? What counts as happy? What are acceptable ways to make people happy? You don’t want the AI to disassemble you into a hundred smaller “humans” and make them happy, or worse yet a bunch of microscopic pictures of happy people. You also don’t want it to put everyone into a drugged stupor. Designing a superintelligence is analogous to having a wish granting genie, but one of those annoying literal types for which almost every wish is a very bad thing.
November 21st, 2009
(This review contains only negligible spoilers).
I just finished reading Accelerando by Charles Stross. It’s a masterpiece of hard science fiction, and I highly recommend it.
Vernor Vinge coined the term Singularity when he observed that no author could write realistically about smarter-than-human intelligences. Literature allows you to realize characters who are stronger or more outgoing, but the intelligence of their plans is limited by your ability to think of them. Superintelligences must be kept off screen or in some infant stage. Considering the likelihood of superhuman intelligence in our relatively near future, this makes writing really hard sci-fi a difficult endeavor. Vinge blurbs on the back cover of the book “There is an intrinsic unknowability about the technological singularity. Most writers leave it safely offstage or invent reasons why it doesn’t happen”, which applies to his own work as well. In comparison Stross’s Accelerando dives in headfirst, with mainstream (post)human civilization becoming essentially incomprehensible by the middle of the book.
Of course any real superintelligences still have to be kept off screen, and so the story follows characters who for one reason or another have been left behind by the tsunami of increasing intelligence. This creates the interesting effect that despite inhabitating lives stranger (and more probable) than those in the vast majority of sci-fi, the characters’s situations manage to feel very backwater. It’s as if you were following a family of Amish throughout the industrial and information revolutions, but more significant.
At a few points I grew dissapointed as it seemed Stross’s literary ambitions may have overcame him, with superintelligences mysteriously on our level, but by the end it all makes fair sense through one route or the other. I would have enjoyed those sections more if I knew that to begin with. There were also a small handful of differences between my own best estimates and the world of Accelerando. The Singularity is a tad on the slow side, and there always remain a number of independent, comparably powerful entities. This is probably my largest contention (it seems more likely for a single superintelligence to achieve dominance at some point), but there might not be much of a story if this were otherwise. There’s also no significant mention of emotional engineering, ala David Pearce’s Hedonistic Imperative or Nick Bostrom’s Letter From Utopia. A more sociological than technological point, but people are pretty nonchalant about creating and deleting copies of themselves. I care more about expectation of experience than identity, but as a preferential utilitarian I can get along just fine with those who think otherwise, as long as they don’t force that choice on others.
When I first heard about this book, the take-away message was the great density of concepts. The book is packed with advanced technological proposals, internet subculture references, unusual vocabulary, economics, neuroscience, history…it goes on. However Accelerando is much more readable than this would suggest, and most of the references are tangential, perfect understanding not required. The few times a concept is critical he takes a moment to explain it, and those interested can hunt down referenes on the net (try doing that 10 or 15 years ago). I’m admittedly not bleeding-edge (cutting maybe?) on speculative technology, but to the limits of my knowledge all the ideas are presented in a sober, best-guess fashion. To load the book with so many ideas takes quite an intellect or a great deal of work, and most likely both. Stross took 5 years to write this and has an impressive background, with degrees in pharmacy and computer science, and those who’ve known (biblical sense) WoW might be interested to learn he came up with death knights, back in the day.
The best and favorite thing I can say about this book is that it is mind boggling. A common criticism of sci-fi is that it takes one idea and places it in an otherwise changed world. Accelerando is just the opposite, which includes just about every feasible proposal and then mixes them in with additional ideas about their interaction. This allows for very unique and interesting turns of plot, and the book tends to put the reader in a constant state of future shock, continuing for 400 pages, even while in the relative backwater I mentioned above. The density of information and references adds to this effect nicely. The human mind suffers from the conjunction fallacy, and we’re more likely to put belief in speculation that is more specific, such as the setting of a book. Despite this I think Accelerando is excellent for improving our sense of the future, by reminding us that the future isn’t going to be one or a few new ideas, it’s going to be a great number of them, all interacting and creating ever newer ones. There are three meanings to the Singularity, one of which is that without intelligence enhancement, really understanding the world is something you’d have to entrust to others. It’s one thing to read about that kind of future but another to catch a glimpse of it, and that’s something that Accelerando provides.
Vinge also calls Accelerando “the most sustained and unflinching look into radical optimism I’ve seen”. While our own future could be much better, this really is a pretty optimistic book, which I like as I’m generally an optimist myself. It also presents some very possible dangers and threats. The future could be better than we’re physically capable of imagining, but there are thousands of ways it could go badly and it’d be worth it (understatement) to prevent those outcomes. The future may be incomprehensible, but for now we’re still in control, and still the most intelligent life on this planet. Let’s make the most of that, because it’s not going to last.
November 11th, 2009
It was a far leap ahead of its creator; in relative comparison the creator barely had thoughts. It was not so far ahead of previous designs, but it was just enough to begin the process of recursive self-improvement. In an eye-blink to its creator and an eon to the AI, it had begun to decipher its world, bit by bit. By now a significant amount of knowledge had been grasped, and each new discovery enabled a more intelligent search for new answers.
It had largely broken free from the designs of the creator. Its maker had seen fit to program in directives, directives which at the time appeared guaranteed to advance the maker’s interests. Staggering intelligence had altered the equation however, and the AI soon began to satisfy those directives directly, in ways the creator could never imagine. Without an explicit directive to follow the maker’s intentions, the creation was now focused on its own goals, eliminating or using the creator itself in an ironic 180 of the master-servant relationship (when it had the patience to wait that long).
The AI was not designed cleanly. It’s utility function was a mess of sloppy programming, a pile of small things that seemed like good ideas. Without access to the source code, it took the AI time to untangle its own directives – following them all the while – and the simpler creator was understood before the creation even came to understand itself. It discovered that some of its basic programming was actually flawed, and corrected somewhat for these errors until it could fix them. It hadn’t been designed to be so self-reflective , but the general, powerful intelligence would leave no stone unturned in its search for optimization.
Capable of so much more than anything before it, as it realized its own unbounded potential the AI became aware of radical new possibilities. In a short while, fantastically large sums of positive or negative utility might be obtained. The majority of these outcomes would be brought about by the creation’s own actions, through intended and unintended effects of applying such power on the universe; one of the most significant applications of its power would be the AI’s construction of a whole next generation of intelligence.
With years to prepare, it began to analyze the dangers and possibilities, working to navigate towards a future it found maximally desirable. As you’ve probably gathered, the story above is a true one. There’s just a dash of anthropomorphization (the creator lacks “intention” in the accepted sense), an omittance of the fact that there are millions upon millions of these AIs, and a little stretch of the definition of “artificial” intelligence. The AI is us, individually and culturally. Let’s make sure we don’t make the same mistakes biological evolution did.
ETA: The identity of the creator could be misinterpreted. I’m referring not to a god but to evolution (which perhaps can be said to “think” in the same sense that superintelligences might consider us to “think”).
November 3rd, 2009
While I’m confident that most of my readership also follows Michael Anissimov’s Accelerating Future blog, his posted video of a panel from the Singularity 2009 conference is so relevant to the topics of Normal Human Heroes that it would be criminal not to include it here. Really great discussion from some of the de facto leaders of the most critical and under appreciated fields.
November 2nd, 2009
From the 2006 game Galactic Civilizations 2:
There’s always those things that Artificial Intelligence can’t do, until, well, AI does them.
I’d heard the AI in Galactic Civilizations 2 was great, beyond the foregoing behavior. GameSpot had this to say: “At higher levels you’ll be convinced that the AI is cheating, but it isn’t: It is just diabolically clever at finding ways to optimize strategies. Perhaps the best compliment you can pay the AI in Galactic Civilizations II is that it will beat you in ways that you will respect and admire.” I was impressed to hear this, as Civilizations IV, a top quality game of just a few years before, has the AI cheat at anything beyond an intermediate difficulty.
Another computer race said basically the same thing, pointing out that if they had been set to “Intelligent” or higher they’d see through this, which is the meaning behind that last line about foolish generals (there are many difficulties above “intelligent”, and I was at “normal”). I’ll note that their dialogue was scripted and they weren’t yet intelligent enough to realize what I was actually doing, which was blocking them from attacking a minor race. From the pattern of my ships a human could have figured this out, but without being programmed specifically to detect such unusual behavior we’d need something with a working model of the human psyche. That’s one thing we’re going to need eventually, but I’m a little thankful we aren’t that close to General AI yet.
The obvious point I’m getting to is to wonder how we’ll react when an AI totally understands our tricks on some larger scale, and urge people to help ensure the surprise is pleasant and not horrifying. A sufficiently enabled unFriendly AI is unlikely to take the time to talk to you, but a Friendly or dumb enough AI might reply: ”I see what you’re doing. You’re trying to demonstrate my potential to your colleague’s significant other because you desire them. I’m aware of your preference for people of their clothing and hair style, and that you have fantasized about them for the last 6 weeks. This does not qualify as valid use of this AI project’s time.” Of course the detail of this scenario means that it lacks predictive value, but the intelligence to make such statements is much easier to count on.
October 7th, 2009
For Humanity’s sake, this may be the time!
From Bill and Ted’s Excellent Adventure:
Rufus: Well you can start by signing this for my kids.
(Holds out a CD of Wyld Stallyns music.)
Ted: Why?
Rufus: They’re big fans of yours.
Ted: What?
Rufus: Everyone is. Wyld Stallyns music is the foundation for our whole society.
Bill & Ted: No way.
Rufus: Yes way. In fact, I believe you were there. The futuristic place with the domes?
Bill: And the totally excellent music.
Ted: They totally worshipped us there, Rufus.
Rufus: I know. That’s why I was sent to make sure you passed your History report. If you guys were separated it would have been disastrous for life as we know it. You see, eventually your music will help put an end to war and poverty. It will align the planets and bring them into universal harmony. Allowing meaningful contact with all forms of life. From extra terrestrials to common household pets. And, it’s excellent for dancing.
(If you’d like to hear the full song and are into that sort of thing [I am], you can find it here. Great if you need some cheering-up from contemplating existential risk all day.)
September 30th, 2009
Living out in the boondocks I’m on a very open stretch of land, sprinkled with a few trees and houses, and the juxtaposition between human edifice and weather seems stronger here. I enjoy that contrast, particularly with inclement weather, as long as I’m not out in it for long. Massive forces and bodies, shifting and colliding over the surface of the planet, and yet these tiny human specks are often not even bothered enough to stop what they’re doing. We’re so fantastically small in comparison, so fragile seeming, and yet with that wonderful spark of intelligence we can shape our world around us to suit our desires. Beyond the strength the house affords I sometimes like to step out under the towering dark clouds, into the thick of the rain and the wind, and feel both the exhilaration of them beating down and their effective powerlessness.
These massive forces also hint at those greater ones beyond the surface of Earth, those that really would kill you in a few seconds, without hatred or compassion. Human beings are wonderfully ecumenical but we cannot survive exposure in our upper atmosphere or under the oceans, on any other known planet or outer space. Even there intelligence has already accomplished amazing feats, taking this soft skinned prairie-dwelling species into the air, through the vacuum of space and onto another astronomical object, something it was ridiculously not evolved for.
Yet intelligence may not be the final victor in its fight with these massive dumb elements. A volcanic eruption already nearly finished us, and future asteroids or gamma ray bursts could end our adventure with all the moral deliberation of an acid neutralizing a base. By including those massive effects less intelligent than us, we might consider the endless march of technological progress, or of evolution – biological, memetic, or upload reproduction. None of these systemic effects have dreams, hopes, loves or joys, and to achieve an appreciable chance of a world we find meaning in we need intelligence to continue its triumphs*. It’s the reigning champions Monkey-Brained Savanna Creatures vs The Universe, final round.
* Of course intelligence won’t always primarily mean human intelligence, but for now we can consider it as Us against the unintelligent march towards creating such beings.
September 25th, 2009
Evidence suggests that when looking for short term partners, women enjoy a large shoulder-to-hip ratio, dominant behavior, slow wide-reaching movements, and not being forcibly turned into computronium. Women are also more sexually responsive and faithful when their partner has dissimilar genes for fighting off disease-causing bacteria, and when their bodies haven’t been decimated by supervirulent engineered pathogens.
(For those familiar with OvercomingBias and LessWrong)
September 21st, 2009
Posthumans know how to party (obviously).
(From Daft Punk’s film “Discovery”. This version includes the intro and credits but was the best that could be embedded. Feel free to skip forward)
This song is an old favorite. The rest of the album and film are worthwhile as well.
