VSnake notes: Хрящик

2013-08-22

В закромах нашлось — DataGristle — интересный инструментарий для анализа данных в CSV файлах, написан на Python:

Usage Scenarios

Operational Diagnostics 1 - a marketing sentiment analysis company uses it for quickly discovering problems in spreadsheets sent to them by their customers. The spreadsheets were often found to be subtly malformed, or had invalid values that could be difficult to find. gristle_determinator was used to quickly sanity-check and find outliers.
Operational Diagnostics 2 - a large data warehousing team uses it whenever their bulk load process breaks on invalid data. Their database's bulkloader does not provide much info in this kind of a case, so they use gristle_freaker to quickly size up the nature of the data in a few problematic columns, gristle_viewer to examine individual records, and gristle_determinator to sanity-check the file structure. This has speed up the problem determination and resolution steps enormously.
Feed Analysis - a large data warehousing team uses it whenever they have new potential data sources to analyze. The gristle_determinator quickly finds data quality issues, and identifies characteristics useful for data modeling. On some large complex feeds, it can sometimes perform 8-20 hours of initial analysis in just five minutes.

What's Included

gristle_determinator - Analyses csv files and prints information about the file structure and each field within it.
gristle_slicer - Selects rows and columns out of csv file.
gristle_freaker - Creates frequency distributions of one or more columns of a csv file.
gristle_viewer - Displays a single record from a csv file organized in two columns, with labels to the left and values to the right.