Not sure how I came across this, but I found a course called Calling Bullshit in the Age of Big Data. Created by biologist Carl Bergstrom and statistician Jevin West, this course goes over how to identify faulty reasoning—bullshit, that is—which can either be intentional or unintentional.
Some interesting things I learned were:
Censoring which is when data is only partially known. The given example is a paper on the lifespan of musicians, where hip-hop artists live shorter than in other genres. Like a lot less.
Now consider this. Rap has only been a thing since the late 70s (like 50 years ago), so many of the founding artists are very much alive, so they can’t be factored into a fatality rate, hence they’re removed (censored) from the data set. After all, DJ Kool Herc—founding father of hip hop—is 65 and alive, and most OG Bronx rappers are around that age. Rappers couldn’t really have a life expectancy higher than that.
How bullshit operates. It’s highly asymmetrical so it takes less energy to make it than to disprove it.
Sloppy statistics. Using the wrong average (mean, mode, median), mixing sample types, correlation vs causation among others.
Bad data visualizations.
What makes the scientific method great.
“Big data” still needs a lot of statistical understanding. And ML can’t make bad science good again. As an example, they show the paper which tried to predict criminals from faces, but that has a lot of problems.
Publication bias was probably my favorite part. It goes into the replication crisis, which is mostly caused by unpublished research. “Publish or perish” is also discussed, along with predatory and hijacked journals, which can be trolled (to humorous effect [PDF])
How to call bullshit. It isn’t a matter of saying “you’re wrong”. You need to do it with humility, actually be right, stick to facts, get to the point, explain well, consider an appropriate medium (social media or journal), understand the field rather than “well, actually” it, remember that more information might not help, not get too carried away and remember that the bullshitter are still human.
In all, it’s a great course which goes over how to call BS. I’m looking forward to their book of the same name coming out some time later year. Along with Humble Pi by Matt Parker, I’m hoping to reccomend this book as a way to improve mathematical thinking.