Bernie speaking clearly about AI makes me feel emotional in a way I rarely am about this sort of thing.
A sketch of a nice model of human values I think I first heard from John Wentworth at Iliad II in August 2025:
We don’t have an explicit utility function, but we have a set of reward signals and pretend that there is some true hidden utility function which we try to model with uncertainty. Then we use our reward signals to update our map of a (fictional) utility function.
This lets us protect against value drift and wireheading by resisting things which would cause our reward signals to give us worse information about our imagined true utility function.
I’ve been happier than usual the past few days in a strange way; strange because I can’t explain what’s different.
I continue to fail to understand why otherwise very smart people are so attached to prerational intuitions about consciousness which don’t seem to come along with independent logical arguments
Specifically the intuition that qualia are irreducible is a priori reasonable but leads to all sorts of paradoxes
And since we have no principled reason to believe it it should be quite cheap to discard
This does not mean there aren’t lots of things about consciousness which I am confused about and find mysterious
Anyway if you think you can explain something I am missing contact me
Listen to Shooglenifty
Never buy an SUV.
I love my wife
When I’m tired I sometimes just sit staring into the distance because in the back of my mind I think I’m waiting for some AI agent doing something
Then I realize I actually don’t have any agents doing anything and I’m not waiting for anything outside of me
Motorcycling is how insecure straight men are allowed to dance
Suppose two beings are omniscient about Earth but not each other. One wants to point out one specific human being on this planet to the other.
They must do this in the minimum number of ASCII characters such that they can be quite confident the other being will know exactly who they are referencing.
No previous common knowledge except common knowledge of the planet Earth and everything that can be logically deduced from that common knowledge.
Now suppose you pick a given person. How would you evaluate whether a proposed string has this property?
If they had a pre-agreed codebook, you’d need roughly log₂(8×10⁹) ≈ 33 bits ≈ 5 ASCII characters to index any human. If they are logically omniscient and omniscient about Earth, but do not have a pre-agreed codebook, how good can they get at minimax? How close to five chars?