This is the title of a new NYT article
reflecting the view from then data trenches. It says “data scientists, according to interviews and
expert estimates, spend from 50 percent to 80 percent of their time mired in
this more mundane labor of collecting and preparing unruly digital data, before
it can be explored for useful nuggets.” One data executive, “whose sensor-filled wristband and software track activity,
sleep and food consumption, and suggest dietary and health tips based on the
numbers,” complains how little
this aspect of data analysis is appreciated by “data civilians.” The solution?
But, of course – (almost) full automation of data collection, an effort
spearheaded by a few promising startups. And how about all the research suggesting
that insight is linked to intuition, and excessive analysis and overthinking – and particularly formal modeling – tend
to suppress these “softer” aptitudes? I suspect most “data scientists” will
hardly worry about this.
