carado.moe

tammy's blog about
AI alignment, utopia, anthropics, and more;

also on twitter, lesswrong, rss, and at

smaller X-risk

a superintelligence killing us all is a superintelligent, very large X-risk.

the superintelligence will tile its values in all directions; not just through space at the speed of light or faster, but also, if it can, by hacking physics and traversing across, for example, worldlines of the quantum many-worlds.

we may be able to create smaller X-risks, that only make us extinct in this timeline, on this earth. there are a few reasons we may want to do this:

• other timelines might have a better shot than us, and us booting a superintelligence may reduce their chances through weird stuff like intertimeline hacking
• to avoid S-risks, including S-risks that may be involved in instrumental cosmic-scale X-risk (maybe superintelligence wants to simulate civilizations in various ways for acausal trade or other acausal weirdness reasons?)
• the next intelligent species on earth is more likely than us to solve alignment before superintelligence, and seems likely enough to be at least a little bit aligned with us (better than cosmic X-risk, at least)
• same as above, but for nearby aliens (whether current or future)

smaller X-risk, where we limit damage to just our civilization, seems harder than tiling the cosmos with paperclips; but at least it might be easier than other plans.

in a similar way, reducing our civilization to ashes without actually becoming extinct might also be a way to get another shot, if we think we're likely to do less badly next time.

remember: this is bigger than all of us. when the fate of the cosmos is at play, we can't afford to be too selfish.