The need was to convert the network stream into clear text http request/responses while doing some decoding of the response body. For instance:
request uri + queryString => response body
- Capture the stream – easy using tcpdump
- Filter the http stream – easy using wireshark with a tcp.port eq 80 filter
- Export http #1. using wireshark file -> export objects -> http. This works fine only for files. It does not work for POST requests. FAIL.
- Using tshark and a combination of -Tfields and -e parameters. Did not worked easily enough even if I suspect it would. FAIL.
- Using tcpflow: tcpflow -r test.pcapng -ehttp. This generates some nice flows but it had some disadvantages: requests and responses are in different files and are flow sorted not time sorted. I think this can be overcome by writting a script which parses: report.xml using something like this. FAIL.
- Final idea was based on pcap2har which parses a .pcap file to a har. Some changes to main.py and voila:
<pre lang="python">
logging.info('Flows=%d. HTTP pairs=%d' % (len(session.flows), len(session.entries)))
for e in sorted(session.entries, key=lambda x: x.ts_start):
if e.request.msg.method == 'GET':
print 'GET', e.request.url
elif e.request.msg.method == 'POST':
print 'POST', e.request.url, urlencode({k: v[0] for k, v in e.request.query.items()})
if e.response.mimeType == 'application/octet-stream':
print decode(e.response.text, options.password)
else:
print 'unknown:', e.response.mimeType, e.response.raw_body_length
print '\n'
#write the HAR file