txt-sushi

txt-sushi is a collection of command line utilities for processing comma-separated and tab-delimited files (AKA flat files, spreadsheets). By focusing exclusively on processing text files with a tabular structure, txt-sushi simplifies common tasks like filtering, joining and transformation that would take some effort to accomplish with a more powerful scripting language.

The source code is available under the BSD open source license.

txt-sushi is still under active development and has not yet reached 1.0. This means that you can expect it to be a little bit rough around the edges but it should generally work as advertised.

Building/Revision Control/Issue Tracking/Announcements:

• txt-sushi can be built with GHC >= 6.10.1 and Cabal. Since txt-sushi is on hackage you should be able to just do:

  cabal update
  cabal install txt-sushi

  Make sure that the cabal bin directory (usually ~/.cabal/bin) is on your PATH
• Revision Control and Issue Tracking: the txt-sushi repository is hosted on github
• Announcements: txt-sushi Announcement RSS feed.

tssql:

The tssql utility allows you to use a subset of the SQL SELECT grammar on your flat files. See the tssql documentation for details.

csvtotab:

Converts a comma-delimited file to a tab-delimited file. This command takes one argument which is either a '-' for standard input or a filename.

Pretty format a comma-delimited file. Formats columns so that all data in column lines up (see examples above). This command takes one argument which is either a '-' for standard input or a filename.

Transpose a CSV file. This command takes one argument which is either a '-' for standard input or a filename.

Converts a tab-delimited file to a comma-delimited file. This command takes one argument which is either a '-' for standard input or a filename.

Pretty format a tab-delimited file. Formats columns so that all data in column lines up. This command takes one argument which is either a '-' for standard input or a filename.

Transpose a tab-delimited file. This command takes one argument which is either a '-' for standard input or a filename.

Adds a header row to a CSV file. Uses column names like col1, col2, col3... This command takes one argument which is either a '-' for standard input or a filename.

Links

There are many other great tools that help you work with flat files. Email me (keithshep@gmail.com) if you think there are other tools that should be added to this list:

Utility	Notes
csvfix	Very similar features to txt-sushi, uses many small utilities rather than SQL syntax
MS Log Parser	Does not join tables, only works with MS Windows, closed source, works with many more data types than just flat files (XML, windows registry ...)
Fsdb	Similar features to txt-sushi but uses many utilities rather than SQL
shsql	Intended to work on mutable persistent CSV data where txt-sushi is more inteded to work on "data streams".
HaExcel	An interesting tool that can be used to intelligently refactor spreadsheets into real databases
MySQL CSV Storage	You can now use CSV files as the storage backend for MySQL