Are we doomed? Are AIs uncontrollable? Yeah. AGI alignment is probably hopeless.
For the last point on AGIs training AGIs, wouldn't the training one have to be aligned? What I mean is that isn't it possible both simply become deceptively aligned while truthfully being wild? In a courtroom, both lawyers try to smear the other appealing to what is right, legal, and true, but they can only do that if they know what is right, legal, and true (aka already aligned). If two lawyers went into a room without any knowledge of the legal system (stand-in for human morality), how would they go about smearing each other? Great article overall tho. I've just started reading ur substack and lots of it is very cool *thumbs up*