Posts Tagged ‘python’

Parsing network stream into http request/response

The need was to convert the network stream into clear text http request/responses while doing some decoding of the response body. For instance:

request uri + queryString => response body

  1. Capture the stream – easy using tcpdump
  2. Filter the http stream – easy using wireshark with a tcp.port eq 80 filter
  3. Export http #1. using wireshark file -> export objects -> http. This works fine only for files. It does not work for POST requests. FAIL.
  4. Using tshark and a combination of -Tfields and -e parameters. Did not worked easily enough even if I suspect it would. FAIL.
  5. Using tcpflow:  tcpflow -r test.pcapng -ehttp. This generates some nice flows but it had some disadvantages: requests and responses are in different files and are flow sorted not time sorted. I think this can be overcome by writting a script which parses: report.xml using something like this. FAIL.
  6. Final idea was based on pcap2har which parses a .pcap file to a har. Some changes to main.py and voila:
logging.info('Flows=%d. HTTP pairs=%d' % (len(session.flows), len(session.entries)))
 
for e in sorted(session.entries, key=lambda x: x.ts_start):
    if e.request.msg.method == 'GET':
        print 'GET', e.request.url
    elif e.request.msg.method == 'POST':
        print 'POST', e.request.url, urlencode({k: v[0] for k, v in e.request.query.items()})
    if e.response.mimeType == 'application/octet-stream':
        print decode(e.response.text, options.password)
    else:
        print 'unknown:', e.response.mimeType, e.response.raw_body_length
    print '\n'
 
#write the HAR file

 

Create a database of exif data

Create a database of exif data from photos using pyexiv2 and save it in a sqlite database for futher query:
(more…)

py-gps-tools

After spending loosing time in vain trying to convert kml files containing the gx:Track format to gpx files for my gps and finally writing my own tool for doing that I realized I have done quite a lot of small scripts for gps data manipulation and decided to push them on github. I started with 2 and as I will clean the others I will push them also.

py-gps-tools

A set of python scripts to manipulate GPS data

kmlCleanTimeStamp.py

Removes TimeStamp information from .kml files. More precisely, it removes gx:TimeStamp and //kml:Placemark/kml:TimeStamp elements. Modifies the document in place.

kmlCleanTimeStamp.py file.kml

kmlTrack2LineString.py

Converts kml gx:Track to old style LineString elements. This is required because not even gpsbabel can convert the gx:Track elements to gpx for instance resulting in data loss. Conversion is done in place.

kmlTrack2LineString.py file.kml

 

Let’s decrypt

AES encrypt in java and decrypt in java, flex, python, C#.

Encrypt: java

public static void encrypt(InputStream is, OutputStream out, String secret) throws Exception {
	SecretKey secretKey = new SecretKeySpec(Hex.decodeHex(secret.toCharArray()), "AES");
        Cipher cipher = Cipher.getInstance("AES/CBC/PKCS5Padding");
        cipher.init(Cipher.ENCRYPT_MODE, secretKey);
        out.write(cipher.getParameters().getParameterSpec(IvParameterSpec.class).getIV());
        CipherOutputStream cipherOutputStream = new CipherOutputStream(out, cipher);
        int bufLength = KEY_LENGTH/8 * 100;
        byte buf[] = new byte[bufLength];
        int bytesRead = 0;
	while((bytesRead = is.read(buf)) != -1 ) {
		cipherOutputStream.write(buf, 0, bytesRead);
	}
	cipherOutputStream.close();
}

(more…)

Skype logs archiving on linux

I keep migrating my Skype logs from installation to installation and they are getting pretty large. As they are binary files there is no easy way to split them properly. However I’ve found this tool which parses the logs and outputs the entries. The output however it’s not that usable. For this reason I’ve wrote a python script which organizes the output from the previous tool and generates files in the form: logs/skype-name/date.log which can be stored, are easy to backup and so on. The code, wrote in between 00:10-01:00 AM last night :), is bellow. Note that this is working with Skype 2.2.0.35 beta on linux. From what I understood it does not applies to newer versions on windows.

To use it you use the following sequence:

./skypelog ~/.Skype/[skype-name]/chatmsg*.dbb > skype.log
./skype-logs.py skype.log [skype-name] logs

(more…)

Cut, reverse and combine GPS tracks

When I plan a cycling trip I search for tracks in the target area and try to maximize the “fun factor”. Thus I often end up with a bunch of tracks from which I want to use parts. The difficult task is to cut, sometime reverse and combine the source tracks into a planned track. Most of the times this seems a too much waste of time since it involves a number of operations in qlandkarte (cutting and combining) and gpsbabel (reversing).

For this purpose here is a python script which takes as input a kml file containing all the tracks and some extra waypoints which define the cutting points then creates a combined track. Here is an example of how some original tracks look in google earth while planning a 2 day cycling trip in Parang mountains:

Original tracks

Original tracks

(more…)

Migrate contacts from C510 to HTC

I order to migrate contacts from a Sony Ericsson C510 to a HTC Desire you first have to export them in an usable format. Wammu recognizes this phone and exports a .vcf file containing all contacts. However there are some major limitations to this process:

(more…)

Searching for duplicate flex functions

After a day lost trying to find a bug caused while refactoring multiple versions of copy/paste code into a component I’ve decided to write a small application which searches for other code from my component still duplicated in other places.
Naturally a basic python script has emerged. It takes as argument a file and a folder and searches for files ending with .as recursively. Both the reference file and the found files are parsed and a basic function reference is created which is compared to the other files.
This method could be easily used for C, Java or other brace based language. The parsing is very basic and some obvious limitations exist such as comment handling. But it’s ok for 10 minutes of work and gave me enough information to simplify my work for the days ahead and also find some interesing informations about the copy-paste anti-pattern.

(more…)

GPS track analysis

GPS data has become extremely available in the last time and analyzing it always yields interesting results.The purpose of this article is to show a simple example where GPS track data is compared with a set of waypoints in order to determine whether a specific tracks was followed. This scenario is custom to various competitions ranging from randonee, cycling (cross country, orientation), 4×4 and the examples can go on and on. A more advanced version of this simple script was used for the TAT competition this year. (more…)

Python xml namespace parsing with libxml2

The goal of this tinkering was simple: to parse a KML file for further interpretation and use using python and libxml2. (more…)