posted on 2023-03-13 — also cross-posted on lesswrong, see there for comments

your terminal values are complex and not objective

a lot of people seem to want terminal (aka intrinsic aka axiomatic) values (aka ethics aka morality aka preferences aka goals) to be simple and elegant, and to be objective and canonical. this carries over from epistemology, where we do favor simplicity and elegance.

we have uncertainty about our values, and it is true that our model of our values should, as per epistemology, generally tend to follow a simplicity prior. but that doesn't mean that our values themselves are simple; they're definitely evidently complex enough that just thinking about them a little bit should make you realize that they're much more complex than the kind of simple model people often come up with.

both for modeling the world and for modeling your values, you should favor simplicity as a prior and then update by filtering for hypotheses that match evidence, because the actual territory is big and complex.

there is no objectively correct universal metaethics. there's just a large, complex, tangled mess of stuff that is hard to categorize and contains not just human notions but also culturally local notions of love, happiness, culture, freedom, friendship, art, comfort, diversity, etc. and yes, these are terminal values; there is no simple process that re-derives those values. i believe that there is no thing for which i instrumentally value love or art, which if you presented me something else that does that thing better, i would happily give up on love/art. i value those things intrinsically.

if you talk of "a giant cosmopolitan value handshake between everyone", then picking that rather than paperclips, while intuitive to you (because you have your values) and even to other humans doesn't particularly track anything universally canonical.

even within the set of people who claim to have cosmopolitan values, how conflicts are resolved and what "everyone" means and many other implementation details of cosmopolitanism will differ from person to person, and again there is no canonical unique choice. your notion of cosmopolitanism is a very complex object, laden with not just human concepts but also cultural concepts you've been exposed to, which many other humans don't share both across time and space.

there is no "metaethics ladder" you can which climb up in order to resolve this in an objective way for everyone, not even all humans — what ladder and how you climb it is still a complex subjective object laden with human concepts and concepts from your culture, and there is no such thing as a "pure" you or a "pure" person without those.

some people say "simply detect all agents in the cosmos and do a giant value handshake between those"; but on top of the previous problems for implementation details, this has the added issue that the things whose values we want to be satisfied aren't agents but moral patients. those don't necessarily match — superintelligent grabby agents shouldn't get undue amounts of power in the value handshake.

some people see the simplicity of paperclips as the problem, and declare that complexity or negentropy or something like that is the ultimate good. but a superintelligence maximizing for that would just fill the universe with maximally random noise, as opposed to preserving the things you like. turns out, "i want whatever is complex" is not sufficient to get our values; they're not just anything complex or complexity itself, they're an extremely specific complex set of things, as opposed to other equally complex sets of things.

entropy just doesn't have much to do with terminal values whatsoever. sure, it has a lot to do with instrumental values: negentropy is the resource we have to allocate to the various things we want. but that's secondary to what it is we want to begin with.

as for myself, i love cosmopolitanism! i would like an egalitarian utopia where everyone has freedom and my personal lifestyle preferences aren't particularly imposed on anyone else. but make no mistake: this cosmopolitanism is my very specific view of it, and other people have different views of cosmopolitanism, when they're even cosmopolitan at all.

see also:

posted on 2023-03-13 — also cross-posted on lesswrong, see there for comments

CC_ -1 License unless otherwise specified on individual pages, all posts on this website are licensed under the CC_-1 license.
unless explicitely mentioned, all content on this site was created by me; not by others nor AI.