How well would you do as an expert?

In the Netherlands (and I’m sure there are versions of it in the UK and the US as well) there is a weekly radio show containing a returning item in which music experts are asked to compare and judge two or three CD recordings of the same piece, without knowing who the musicians are. They have to guess the performers and describe why they do (or don’t) like that particular performance.

How well would you do in such a test? The common hypothesis is that experts do this much better, e.g. under the assumption that they are more sensitive in their listening skills. But do experts indeed hear more detali and more nuances when compared to a 'common listener'? Or do they just have more terminology available to verbalize these differences?

Two years ago our group did a large-scale online listening experiment with a similar task. Participants were asked to compare several pairs of recordings of well-known musicians. One of the recordings was taken directly from a CD, but the other was originally performed at another tempo (faster or slower) and then scaled to be similar in tempo to the former recording. The task was to judge which recording was real and which one was manipulated, by focusing on the timing used by the performer.

To give you an idea of the difficulty of the task, below an example.

A
B
(See answer at the bottom.)

The results were recently published in the Journal of Experimental Psychology, with a surprising outcome: the judgments seem to be largely influenced by exposure to music (listening a lot to one’s favorite music) and not (at all) by the level of expertise (amount of formal musical training). One seems to learn a lot by simply listening.

Honing, H., & Ladinig, O. (2009). Exposure influences expressive timing judgments in music. Journal of Experimental Psychology: Human Perception and Performance, 35 (1), 281-288 DOI: 10.1037/a0012732

* The first recording is the original. It is Glenn Gould performing English Suite No. 4 by J.S. Bach. The second recording is Sviatoslav Richter performing the same piece. However, this recording was sped up from 70 to 87 bpm making his use of tempo rubato 'unnatural'.

What makes a theory compelling?*

Karl Popper was a philosopher of science that was very much interested in this question. He tried to distinguish 'science' from 'pseudoscience', but got more and more dissatisfied with the idea that the empirical method (supporting a theory with observations and experiments) could effectively mark this distinction. He sometimes used the example of astrology “with its stupendous mass of empirical evidence based on observation”, but also nuanced it by stating that “science often errs, and that pseudoscience may happen to stumble on the truth.”

Next to his well-known work on falsification, Popper started to develop alternatives to determine the scientific status or quality of a theory. He wrote the complex yet intriguing sentence “confirmations [of a theory] should count only if they are the result of risky predictions; that is to say, if, unenlightened by the theory in question, we should have expected an event which was incompatible with the theory — an event which would have refuted the theory.” (Popper, 1963).

Popper was especially thrilled with the result of Eddington’s eclipse observations, which in 1919 brought the first important confirmation of Einstein's theory of gravitation. It was the surprising consequence of this theory that light should bend in the presence of large, heavy objects (Einstein was apparently willing to drop his theory if this would not be the case). Independent of whether such a prediction turns out to be true or not, Popper considered it an important quality of ‘real science’ to make such ‘risky predictions’. Interesting thought, not?

I still find this an intriguing idea. The notion of ‘risky’ or ‘surprising predictions’ might actually be the beginning of a fruitful alternative to existing model selection techniques, such as goodness-of-fit (which theory predicts the data best) and simplicity (which theory gives the simplest explanation). Also in music cognition measures like goodness-of-fit (r-squared, percentage variance accounted for, and other measures from the experimental psychology toolkit) are often used to confirm a theory. Nevertheless, it is non-trivial to think of theories that make surprising predictions. That is, a theory that predicts a yet unknown phenomenon as a consequence of the intrinsic structure of the theory itself. If you know of any, let me know!

K. R. Popper (1963). Conjectures and Refutations. London: Routledge.

* Repeated blog entry from July 23, 2007 (celebrating finalizing a research proposal with Jan-Willem Romeijn on these issues, hoping to be able to address these issues head-on ;-)