I’ve been thinking a bit more about the comment I made yesterday that there used to be a time that physics discoveries were made by people but now we just need to build a machine to do it (the LHC).
The major science discoveries, almost by definition, are unexpected and can be very serendipitous. The discoverer wasn’t out there looking for something new per se, but, through some lucky sequence of events, it presented itself to him (and, unfortunately, most physics discoveries are still ‘him’ not ‘her’). Three examples spring to mind: Oersted’s discovery that an electric current produces a magnetic field, Becquerel’s discovery of radioactivity, and Geiger/Marsden/Rutherford’s discovery that an atom has a nucleus.
In Oersted’s case, so the story goes, he was giving a lecture (1820) on this new ‘electricity’ thing, and just happend to have a compass sitting on his bench. He saw that when the current was switched on, the compass needle moved. The skill of the physicist here was to realize that this phenomenon was something that should be investigated further, which Oersted duly did, to great success.
In Becquerel’s case (1896), he found that unexposed photographic plates had somehow become exposed. Rather than waving this away and thinking that somehow light must have leaked in, he tracked it down to the fluorescent uranium salts that he was preparing to experiment on. Yes, Becquerel got lucky, but it was his careful experimenting afterwards that resulted in him recognizing that something ‘new’ was happening.
One could argue that the discovery of the atomic nucleus was less down to luck – after all, Geiger and Marsden were carrying out an experiment (in 1909) to look at the properties of alpha particles (i.e. discover ‘new’ knowledge about this particle) - but what they got was completely unexpected. Again, careful consideration and analysis of the phenomenon by Rutherford led him to conclude that the positive charge in the atom must be concentrated in a nucleus – a previously unknown piece of physics.
None of these steps forward in physics was ‘expected’ (though in Oersted’s case there was already some suspiscion of a link between electricity and magnetism). Yes, to some extent, these physicists created their own luck, and certainly exercised skill in recognizing and interpreting what they saw, but, fundamentally, they weren’t expecting to see something sensational. Contrast this with the LHC, where something entirely new (but, of course, we don’t quite know what yet) is anticipated. A machine to do physics for us.

Great post. I was at a meeting of chemistry educators yesterday and a university lecturer pointed out that the best PhD students are those with great observational skills.
I think the big breakthroughs often require a bit of luck but, they also rely on
1) someone being observant enough to notice that something different or unusual is occurring.
2) and then working out a way to explain/test/apply the new phenomenon.
“Luck is what happens when preparation meets opportunity” Seneca
“I’m a great believer in luck, and I find the harder I work the more I have of it” Thomas Jefferson
There was an interesting NZ programme a few years ago that took a group of New Zealanders and did various psychological tests on them. One of the things they found was that those who considered themselves to be “lucky” were those who, when tested, were very observant. For example, in one test the “lucky” subjects when following a specified route in the street were much more likely to spot money and other valuable placed along their route.