Parsing network stream into http request/response

The need was to convert the network stream into clear text http request/responses while doing some decoding of the response body. For instance:

request uri + queryString => response body

  1. Capture the stream – easy using tcpdump
  2. Filter the http stream – easy using wireshark with a tcp.port eq 80 filter
  3. Export http #1. using wireshark file -> export objects -> http. This works fine only for files. It does not work for POST requests. FAIL.
  4. Using tshark and a combination of -Tfields and -e parameters. Did not worked easily enough even if I suspect it would. FAIL.
  5. Using tcpflow:  tcpflow -r test.pcapng -ehttp. This generates some nice flows but it had some disadvantages: requests and responses are in different files and are flow sorted not time sorted. I think this can be overcome by writting a script which parses: report.xml using something like this. FAIL.
  6. Final idea was based on pcap2har which parses a .pcap file to a har. Some changes to and voila:'Flows=%d. HTTP pairs=%d' % (len(session.flows), len(session.entries)))
for e in sorted(session.entries, key=lambda x: x.ts_start):
    if e.request.msg.method == 'GET':
        print 'GET', e.request.url
    elif e.request.msg.method == 'POST':
        print 'POST', e.request.url, urlencode({k: v[0] for k, v in e.request.query.items()})
    if e.response.mimeType == 'application/octet-stream':
        print decode(e.response.text, options.password)
        print 'unknown:', e.response.mimeType, e.response.raw_body_length
    print '\n'
#write the HAR file


Leave a Reply