["python","parsing"]
How to efficiently parse fixed width files?

I am trying to find an efficient way of parsing files that holds fixed width lines. For example, the first 20 characters represent a column, from 21:30 another one and so on. Assuming that the line holds 100 characters, what would be an efficient way to parse a line into several components? I could use string slicing per line, but it's a little bit ugly if the line is big. Are there any other fast methods?
Using the Python standard library's struct module would be fairly easy as well as fairly fast since it's written in C. The code below how it use it. It also allows columns of characters to be skipped by specifying negative values for the number of characters in the field. import struct fieldwidths = (2, -10, 24) fmtstring = ' '.join('{}{}'.format(abs(fw), 'x' if fw < 0 else 's') for fw in fieldwidths) # Convert Unicode input to bytes and the result back to Unicode string. unpack = struct.Struct(fmtstring).unpack_from # Alias. parse = lambda line: tuple(s.decode() for s in unpack(line.encode())) print('fmtstring: {!r}, record size: {} chars'.format(fmtstring, struct.calcsize(fmtstring))) line = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789\n' fields = parse(line) print('fields: {}'.format(fields)) Output: fmtstring: '2s 10x 24s', recsize: 36 chars fields: ('AB', 'MNOPQRSTUVWXYZ0123456789') Here's a way to do it with string slices, as you were considering but were concerned that it might get too ugly. It is kind of complicated and speedwise it's about the same as the version based the struct module — although I have an idea about how it could be sped up (which might make the extra complexity worthwhile). See update below on that topic. from itertools import zip_longest from itertools import accumulate def make_parser(fieldwidths): cuts = tuple(cut for cut in accumulate(abs(fw) for fw in fieldwidths)) pads = tuple(fw < 0 for fw in fieldwidths) # bool values for padding fields flds = tuple(zip_longest(pads, (0,)+cuts, cuts))[:-1] # ignore final one parse = lambda line: tuple(line[i:j] for pad, i, j in flds if not pad) # Optional informational function attributes. parse.size = sum(abs(fw) for fw in fieldwidths) parse.fmtstring = ' '.join('{}{}'.format(abs(fw), 'x' if fw < 0 else 's') for fw in fieldwidths) return parse line = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789\n' fieldwidths = (2, -10, 24) # negative widths represent ignored padding fields parse = make_parser(fieldwidths) fields = parse(line) print('format: {!r}, rec size: {} chars'.format(parse.fmtstring, parse.size)) print('fields: {}'.format(fields)) Output: format: '2s 10x 24s', rec size: 36 chars fields: ('AB', 'MNOPQRSTUVWXYZ0123456789') Update As I suspected, there is a way of making the string-slicing version of the code faster — which in Python 2.7 make it about the same speed as the version using struct, but in Python 3.x make it 233% faster (as well as the un-optimized version of itself which is about the same speed as the struct version). What the version presented above does is define a lambda function that's primarily a comprehension that generates the limits of a bunch of slices at runtime. parse = lambda line: tuple(line[i:j] for pad, i, j in flds if not pad) Which is equivalent to a statement like the following, depending on the values of i and j in the for loop, to something looking like this: parse = lambda line: tuple(line[0:2], line[12:36], line[36:51], ...) However the latter executes more than twice as fast since the slice boundaries are all constants. Fortunately it relatively easy to convert and "compile" the former into the latter using the built-in eval() function: def make_parser(fieldwidths): cuts = tuple(cut for cut in accumulate(abs(fw) for fw in fieldwidths)) pads = tuple(fw < 0 for fw in fieldwidths) # bool flags for padding fields flds = tuple(zip_longest(pads, (0,)+cuts, cuts))[:-1] # ignore final one slcs = ', '.join('line[{}:{}]'.format(i, j) for pad, i, j in flds if not pad) parse = eval('lambda line: ({})\n'.format(slcs)) # Create and compile source code. # Optional informational function attributes. parse.size = sum(abs(fw) for fw in fieldwidths) parse.fmtstring = ' '.join('{}{}'.format(abs(fw), 'x' if fw < 0 else 's') for fw in fieldwidths) return parse