I preordered my copy.

Something about the tone of this announcement feels very wrong, though. You cite Rob Bensinger and other MIRI staff being impressed. But obviously, those people are highly selected for already agreeing with you! How much did you engage with skeptical and informed prereaders? (I'm imagining people in the x-risk-reduction social network who are knowledgeable about AI, acknowledge the obvious bare-bones case for extinction risk, but aren't sold on the literal stated-with-certainty headline claim, "If anyone builds it, everyone dies.")

If you haven't already done so, is there still time to solicit feedback from such people and revise the text? (Sorry if the question sounds condescending, but the tone of the announcement really worries me. It would be insane not to commission red team prereaders, but if you did, then the announcement should be talking about the red team's reaction, not Rob's!)

Low-IQ voters can't identify good policies or wise politicians; democracy favors political actors who can successfully propagandize and mobilize the largest number of people, which might not correspond to good governance. A political system with non-democratic elements that offers more formalized control to actors with greater competence or better incentives might be able to choose better policies.

I say "non-democratic elements" because it doesn't have to be a strict binary between perfect democracy and perfect dictatorship. Consider, e.g., how the indirect election of U.S. Senators before the 17th Amendment was originally intended to make the Senate a more deliberative body by insulating it from the public.

(Maybe that's all wrong, but you asked "what's the model", and this is an example model of why someone might be skeptical of democracy for pro-social structural reasons rather than just personally wanting their guy to be dictator.)

From "But Hurting People Is Wrong":

Look across the superclusters, and most entities either don't do natural-number arithmetic at all, like stars and rocks; or they do it perfectly up to the limits of bounded cognition, like galaxy-spanning superintelligences. If there's anything odd about humans, it's the way that humans are only halfway finished being sucked into attractors like that.

Best wishes, Less Wrong Reference Desk

(Previous commentary and discussion.)

I'm again not sure how far this generalizes, but among the kind of men who read Less Wrong (which is a product of both neurotype and birth year), I think there's a phenomenon where it's not a matter of a man being cognitively unable to pick up on women's cues, but of not being prepared to react in a functional way due to having internalized non-adaptive beliefs about the nature of romance and sexuality. (In a severe case, this manifests as the kind of neurosis described in Comment 171, but there are less severe cases.)

I remember one time from my youth where a woman was flirting with me in an egregiously over-the-top way that was impossible to not notice, but I just—pretended to ignore it? Not knowing what was allowed, it was easier to just do nothing. And that case was clearly not a good match, but that's not the point—I somehow didn't think through the obvious logic that if "yang doesn't step up", then relationships just don't happen.

Not sure how much this generalizes to everyone, but part of the story (for either the behavior or the pattern of responses to the question) might that some people are ideologically attached to believing in love: that women and men need each other as a terminal value, rather than just instrumentally using each other for resources or sex. For myself, without having any particular empirical evidence or logical counterargument to offer, the entire premise of the question just feels sad and gross. It's like you're telling me you don't understand why people try to make ghosts happy. But I want ghosts to be happy.

No one with the money has offered to fund it yet. I'm not even sure they're aware this is happening.

Um, this seems bad. I feel like I should do something, but I don't personally have that kind of money to throw around. @habryka, is this the LTFF's job??

(I gave $2K.)

Simplicia: But how do you know that? Obviously, an arbitrarily powerful expected utility maximizer would kill all humans unless it had a very special utility function. Obviously, there exist programs which behave like a webtext-next-token-predictor given webtext-like input but superintelligently kill all humans on out-of-distribution inputs. Obviously, an arbitrarily powerful expected utility maximizer would be good at predicting webtext. But it's not at all clear that using gradient descent to approximate the webtext next-token-function gives you an arbitrarily powerful expected utility maximizer. Why would that happen? I'm not denying any of the vNM axioms; I'm saying I don't think the vNM axioms imply that.

(Self-review.) I think this pt. 2 is the second most interesting entry in my Whole Dumb Story memoir sequence. (Pt. 1 deals with more niche psychology stuff than the philosophical malpractice covered here; pt. 3 is a more of a grab-bag of stuff that happened between April 2019 and January 2021; pt. 4 is the climax. Expect the denouement pt. 5 in mid-2025.)

I feel a lot more at peace having this out there. (If we can't have justice, sanity, or language, at least I got to tell my story about trying to protect them.)

The 8 karma in 97 votes is kind of funny in how nakedly political it is. (I think it was higher before the post got some negative attention on Twitter.)

Given how much prereading and editing effort had already gone into this, it's disappointing that I didn't get the ending right the first time. (I ended up rewriting some of the paragraphs at the end after initial publication after it didn't land in the comments section the way I wanted it to land.)

Subsection titles would have also been a better choice for such a long piece (which was rectified for the publication of pt.s 3 and 4); I may still yet add them.