carado.moe

tammy's blog about
AI alignment, utopia, anthropics, and more;

also on twitter, lesswrong, rss, and at

epistemic range

there is a security mindset-ish general principle, of which motivated stopping and motivated continuation are ordinary paranoia-ish special cases, which i call "epistemic range".

motivated stopping and motivated continuation are heuristics that catch some failure modes where you pursue an epistemic investigation — an instance of reasoning about a question in order to improve your belief state about the answer — to whichever extent lets you get a belief state that is the one you want to get. of course, in epistemic rationality, you should not want to believe any particular thing; you want your belief state to correspond to whatever is actually true.

and, that there can be such a thing as {stopping an epistemic inquiry too early} or {continuing an epistemic inquiry for too long} imply that there is a range, and maybe even a particular point, at which you should stop. in this respect, i believe epistemology to be akin to science: you would want to (do the kind of thing that is equivalent to) preregister epistemic investigations with a method for knowing when to stop.

personally, i believe that a good rule of thumb for when to stop is when it feels like you can just as easily come up with narratives for multiple mutually incompatible possibilities. on dath ilan, they probably have a more robust notion of where to stop, and look at the belief state, and decide that this is what you believe for now until you either

• get more object-level evidence about the topic
• get more meta-level evidence about where your stopping point should be
• get more contextual evidence about how much confidence you need in this belief, to accomplish your goals

which occur to me as the three main things that impact how far one should want to pursue an epistemic investigation.

note that epistemic range depends, among other things, on where your epistemic investigation has gotten you; as a mathematical function, epistemic range should be a function from current-state-of-epistemic-investigation to boolean (whether to stop or not), not a function from question and context to static number.

on a particular subject matter, the notion of {how far you should go} is your epistemic range. the set of all your epistemic ranges on various questions is your epistemic frontier, and it nicely draws a shape representing how much you can figure out.