Big Data - do we have the skills for this?

Work with health care providers includes repeated discussions of the perceived need to use some Big Data methods to augment the analytics program.  In a sprint for quick results and reduced risk, packaged vendor solutions are attractive.  Once a packaged Big Data service or application is selected and deployed, there's often a moment where the team doing the hands-on work expresses some surprise.  That's when the actual download of an important data set is finally available.  And the result of quite a lot of extracting, staging and filtering is seen to be ... rows and columns.

It's often true that without Hadoop and a brace of other technologies, the required data set would not be available.  And it is well worth remembering that the majority of custom analytics data sets will arrive as a spreadsheet file, or be most easily imported to a desktop spreadsheet tool for final evaluation and display.