Wednesday, May 11, 2011

Breaking up large files

I have been running into tables that are too big for Excel and needed an easy way to break them into smaller chunks. Unix to the rescue on that. If you have a Unix box like Ubuntu or emulator like Cygwin all you need to do is run the command line code split option filename prefix. Split has two options -l (by line number) or -b (size in bytes, I think).

So I had the giant table in a csv, I ran "split -l 500000 d:\folder\folder\bigfile.csv smallerfile" and my big csv was broken up into 500k line csvs. From there I just opened them in a text editor and copied the header line into each new file. 

No comments:

Post a Comment