Ultimately, the goal of superintelligent AI should be to enable humanity to flourish, but defining what that means is incredibly challenging due to the complexity of human values. The way people talk about AI alignment as the end goal worries me; it should be about creating a robust agent assistant instead.