(I have updated this post with a link below to my essay on Medscape.)
No data is better than bad data.
The ProPublica Surgeon Scorecard is not ready for prime time.
It was a good effort. I support investigative journalism and ProPublica. We need more transparency in Medicine. I despise the utter lack of meritocracy in healthcare.
But that doesn’t mean we should embrace flawed data.
My friend, electrophysiologist Dr. Jay Schloss, has written a detailed review of the criticism of the ProPublica project on his blog, Left to My Own Devices.
Jay is one of the sanest, smartest physician voices on the Internet. Here is an excerpt.
So is Surgeon Scorecard bad data? Strong words, but I say yes. This analysis was a great idea, but it fails to deliver on its goals. The data and methodology both have significant flaws. I say that from the perspective of a working clinician and clinical researcher with over 20 years experience. This project is as much science as it is journalism. I therefore see no reason this work should not be peer-reviewed and discussed as would any scientific outcomes study. As I suggested to ProPublica, we need to kick the tires.
Please read Jay’s piece. Then think a bit about data and methods. For instance, if you like your conclusion, that surgeons vary in skill, it’s easy to see past flaws in the experiment.
You can read my take on the Surgeon Scorecard on Medscape: Failing Grade for ProPublica’s Surgeon Scorecard