carado.moe

tammy's blog about
AI alignment, utopia, anthropics, and more;

also on twitter, lesswrong, rss, and at

alignment research is very weird

on its face, AI alignment is just the field of study of how we make AI do what we want. seems simple enough.

in practice, it leads to many very strange places. turns out, making an AI that optimizes anything at all, we don't care what, is much easier than making it robustly optimize for what we want (whatever that is). here are some weird questions that come up and seem like they might actually need figuring out in order to build aligned AI:

• can you control the past?
• how to avoid demons in the space of hypotheses?
• how to generalize bayesian thinking to anthropics?
• are minimal circuits demon-free?
• is the cosmos a graph of causal universe-bubbles? and how are properties inherited between those?
• could alignment research be creating even more risk?
• when do identical/similar computations get more "ethics juice"?
• are there finitely many moral patients? which ones are viable? what the hell even is a moral patient?
• is BQP=P? can we resimulate earth?
• on what levels are we many-instanced?
• what's a good decision theory which can reason about its embeddedness?
• how are we to care about existential self-determination?
• in what sense, if any, is our civilization quantum-immortal?
• what kind of utopia could we be aiming at?
• how much compute can we eventually grab?
• and many others.

it may be that some or most of those questions are irrelevant; for example, it may be that we can just build "dumb AI" that's limited in scope to writing poetry and designing non-smart-AI software, and somehow everyone agrees to only make that kind of AI (as opposed to facebook AI killing everyone six months later). but for the general case where AI is supposed to be arbitrarily capacitous, in a way that most AGI labs are pursuing (sometimes even intentionally and explicitely) these questions are relevant — at the very least in the meta sense of "which questions do we actually need to figure out".