carado.moe

tammy's blog about
AI alignment, utopia, anthropics, and more;

also on twitter, lesswrong, rss, and at

making decisions as our approximately simulated selves

an agent should want to realize their values, and in particular should want their approximated selves — as guessed about by a smart oracle — to also make the decisions that realizes their values. for example, in newcomb's problem, you want omega's guess of you to maximize how much you actually get from the entire problem.

now, imagine that you're told that you're not the "real" you, you're the simulated you inside omega. and you're not even being simulated to a very high level of detail, you're instead an approximated simulation (AS). you should want to accept this, of course — just like you should want to rule out materially acausal things even when you get a very strong intuition about them, you should want to rule out even the possibility that anything you're percieving is actually happening, and instead simply roll with it and say "well, i'll definitely one-box then".

i think this reasoning should reasonably extend to implementing your values in general, even if your values entail not caring about things that are sufficiently not moral patients and if the AS-you is in fact simulated at a low level enough of detail to not count as a moral patient. if you and some AS-you have to decide which one of you and AS-you will experience some suffering, both of you's should decide it should be AS-you — or in other words, you should have a decision theory that is ready to say "yeah, i'm okay with undergoing suffering, because i think that i'm only an AS and not the full me that my values care about".

which is a perhaps unintituive result! but it does make sense — after all, a character in fiction can make decisions, but we don't believe it generally counts enough as a moral patient that we would effectively care if it suffers. this is a similar situation, but as if we reflected about the simulation from inside the work of fiction — and we should be the kind of agent which comes to the globally correct decision even if we notice that we're in a weird part of it, such as being inside omega's prediction or being inside fiction.