Sabine Hossenfelder: Backreaction: An update on the status of superdeterminism with some personal notes

In December I put out a video on superdeterminism that many of you had asked for. I hesitated with this for a long time. As you have undoubtedly noticed, I don’t normally do videos about my own research. This is because I can’t lay out all the ifs and buts in a 10 minutes video, and that makes it impossible to meet my own scientific standards. I therefore eventually decided to focus the video on the most common misunderstandings about superdeterminism, which is (a) that superdeterminism has something to do with free will and (b) that it destroys science. I sincerely hope that after my video we can lay these two claims to rest.

A person by the name of Bernado Kastrup criticized my research on the basis of this video. He demanded that we talk about his criticism. I initially ignored him because of several reasons.

He got things wrong as soon as he started writing, showing that he either didn't read my papers or didn't understand them. A lot of people pick on me because they want to draw attention to themselves and that's a game I'm not willing to take part in.

Third, Kastrup has written a bunch of essays about consciousness and something-with-quantum and “physicalism” which makes him the kind of person I generally want nothing to do with. Just to give you an idea, let me quote from one of his essays:

“Ordinary phenomenal activity in cosmic consciousness can thus be modelled as a connected directed graph. See Figure 1a. Each vertex in the graph represents a particular phenomenal content and each edge a cognitive association logically linking contents together.”

And here is the figure: 

I didn't want to talk to him because I had read what he had written, but I hadn't accused him of anything.

The fourth and final reason that I didn’t want to talk to him is that I get a lot of podcast request, and I don’t reply to most of them simply because I don’t have the time.

I consulted with some friends and decided to talk to Kastrup anyway. I'd spoken with the host of the discussion and I like him. He is a smart young man and if you don't take away anything else from this post, then at least check out his YouTube channel, which is worth some time. I thought that Kastrup's understandings might help other people.

A week later, the only good thing I can report about my conversation with Kastrup is that he didn’t bring up free will, which I think is progress. Unfortunately, he didn’t seem to know much about superdeterminism even after having had time to prepare. He eventually ran out of things to say and then accused me of being “combative” after clearly being surprised to hear that an interaction with a single photon isn’t a measurement. Srsly. Go listen to it.
Instead of concluding that he’s out of his depth, he then wrote another blogpost in which he accused me of “misleading, hollow, but self-confident, assertive rhetoric”, claimed that “Sabine has a big mouth and seems to be willing to almost flat-out lie in order to NOT look bad when confronted on a point she doesn’t have a good counter for.” And, “Her rhetorical assertiveness is, at least sometimes, a facade that hides a surprising lack of actual substance.” Keep in mind that this is a person who claimed to “model” the “phenomenal activity in cosmic consciousness” with 16 circles. Speak of lack of substance. Lesson learned: I was clearly too optimistic about the possibility of rational discourse, and don’t think it makes sense to further communicate with this person. Having said that, I gather that some people who watched the exchange were genuinely interested in the details, so I want to add some explanations that didn’t come across as clearly as I hoped they would. First of all, the reason I am interested in superdeterminism has nothing whatsoever to do with physicalism or realism (I don’t know what these words mean to begin with). It’s simply that the collapse postulate in quantum mechanics isn’t compatible with general relativity because it isn’t local. That’s a well-defined mathematical problem and solving such problems is what I do for a living. Note that simply declaring that the collapse isn’t a physical process doesn’t explain what happens and hence doesn’t solve the problem. We need to have some answer for what happens with the expectation value of the stress-energy-tensor during a measurement. I’m an instrumentalist; I am looking for a mathematical prescription that reproduces observations, one of which is that the outcome of a measurement is a detector eigenstate. The obvious solution to this problem is that the measurement process which we have in quantum mechanics is an effective, statistical description of an underlying local process in a hidden variables theory. We know from Bell’s theorem (or its observed violations, respectively) that if a local theory underlies quantum mechanics then it has to violate statistical independence. That’s what is commonly called “superdeterminism”. In such theories the wave-function is an average description, hence not “real” or “physical” in any meaningful way.

Why am I interested in superdeterminism? General relativity is local. It's beyond me why everyone else wants to hold onto an assumption that is problematic and unjust, and is willing to throw out locality, but that's the situation.

Now, the variables in this yet-to-be-found underlying theory are only “hidden” in so far as that they don’t appear in quantum mechanics; they may well be observable with suitable experiments. This brings up the question what a suitable experiment would be. It is clear that Bell-type tests are not the right experiments, because superdeterministic theories violate Bell inequalities just like quantum mechanics. In fact, superdeterministic theories, since they reproduce quantum mechanics when averaged over the hidden variables, will give the same inequality violations and obey the same bounds as quantum mechanics. (Some people seem to find this hard to understand and try to impress me by quoting other inequalities than Bell’s. You can check for yourself that all those inequalities assume statistical independence, so they cannot be used to test superdeterminism.)

This is why I wrote a paper in which I proposed a mostly model-independent test for hidden variables. Chris Fuchs told me that von Neumann made a proposal 50 years ago, but the experiment was never done. It hasn't been done.

A key point of the 2011 paper was that one does not need to make specific assumptions about the hidden variables. One reason I did this is that Bell’s theorem works the same way: you don’t need to know just what the hidden variables are, you just need to make some assumptions about their properties.

In my book Lost in Math, I explain that math alone isn't enough to develop a new theory. Data is needed to develop the hidden variables theory. The chance that any one of the models is correct is basically zero without that.

This is why I did not want to develop a model for the hidden variables – it would be a waste of time. It didn’t work for phenomenology beyond the standard model and it won’t work here either. Instead, we have to identify the experimental range where evidence could be found, collect the data, and then develop the model. Unfortunately and, in hindsight, unsurprisingly, the 2011 paper didn’t go anywhere. I think it’s just too far off the current mode of thinking in physics, which is all about guessing models and then showing that those guesses are wrong, a methodology that works incredibly badly. Nevertheless, I have since spent some time on developing a hidden variables model, but it’s going slowly, partly because I think it’s a waste of time (see above), but also because I am merely one person working four jobs while raising two kids and my day only has 24 hours.

However, in contrast to what Kastrup accuses me of, I have repeatedly and clearly stated that we do not have a satisfactory superdeterministic hidden variables model at the moment. I say this in pretty much all of my talks, it’s explicitly stated in my paper with Tim (“These approaches [...] leave open many questions and it might well turn out that none of them is the right answer.”). I also said this in my conversation with Kastrup. 

I want to stress that the reason I didn't write down a particular hidden variables model is not that it can't be done, but that there are too many ways it could be done.

Next thing he got confused about is that two years ago, Sandro an I cooked up a superdeterministic toy model. The point of this model was not to say that it should be experimentally tested. We merely put this forward to demonstrate once and for all that superdeterministic models do not require “finetuning” or any “conspiracies”.

The toy model reproduces quantum mechanics exactly, but it is local and deterministic on the expense of violating statistical independence. The claim that superdeterminism ruins science is wrong since it reproduces quantum mechanics.

But besides this, it is a rather pointless and ad hoc toy model that I don’t think makes a lot of sense for a number of reasons (which are stated in the paper). Still, it demonstrates that of course if you want to then can define your hidden variables somehow. I should also mention that our model is certainly not the first superdeterministic hidden variables model. (See references in paper.) There are a lot of toy models in quantum foundations like this with the purpose of shedding light on one particular assumption or another, and my model falls in this tradition. I could easily modify this model so that it would make predictions that deviate from quantum mechanics, so that one could experimentally test it. But the predictions would be wrong, so why would I do this. Having said that, my thinking about superdeterminism has somewhat changed since 2011. I was at the time thinking about the hidden variables the way that they are usually portrayed as some kind of extra information that resides within particles. I have since become convinced that this doesn’t work, and that the hidden variables are instead the degrees of freedom of the detector. If that is so, then we do know what the hidden variables are, and we can estimate how likely they are to change. Hence, it becomes easier to test the consequences. This is why in my later papers and in my more recent talks I mention a simpler type of experiment that works for this case – when the hidden variables are the details of the detector – specifically. I have to stress though that there are other models of superdeterminism which work differently and that can’t be tested this way.

I don't know what the evolution law looks like. I think it can be done with a differential equation. I wrote a paper with Sandro that proposed a new path-integral formalism that could incorporate the required type of evolution law. It was accepted for publication the other day.

What we do in the paper is to define the formalism and show that it can reproduce quantum mechanics exactly – a finding I think is interesting in and by itself. As Kastrup said entirely correctly, there are no hidden variables in that paper. I don’t know why he thought there would be. The paper is just not about hidden variables theories.  I have a number of ideas of how to include the detector degrees of freedom as hidden variables into the path integral. But again the problem isn’t that it can’t be done, but that there are too many ways it can be done. And in none of the ways I have tried can you still calculate something with the integral. So this didn’t really go anywhere – at least so far. It doesn’t help that I have no funding for this research.

I think the best way to move forward is to experiment with a small, cold system and see if there are any deviations from quantum mechanics. If anyone is interested in helping with the path integral, please shoot me a note because I have a lot more to say.

Finally, I have given a number of seminars about superdeterminism, at least one of which is on YouTube here. I also some months ago did a discussion with Matt Leifer which is here. Leifer, I would say, is one of the leading people in the foundations of quantum mechanics at the moment. I may have some disagreements with him but he knows his stuff. You will learn more from him than from Kastrup.

The brief summary is for those who jumped over some of the more complex paragraphs. If you join people like Kastrup who complain about physicalism, you will have to accept that detectors can both click and not click at the same time.

I have concluded that quantum mechanics is not a fundamental theory. We need to find the right experiment to understand the underlying physics.