How can I get python requests to act more like curl?

  curl, http, python, python-requests, tcpdump

I am trying to write a python module to communicate with a fixed HTTP server on a hardware device in order to send data to it. I am able to send data correctly via curl, but for some reason it does not work correctly when I use the requests module in python.

I have confirmed (by using httpbin.org/post) that the two requests are identical, but for some reason only the one send via curl actually works.

When I look at the tcpdumps of the two requests, I do see a difference: The initial handshake is essentially identical, and then the data is sent (in both cases) as three separate packets.

From curl, the communication post-handshake looks like:

17:58:31.691251 IP CLIENT.56184 > SERVER.http: Flags [P.], seq 1:232, ack 1, win 29200, length 231: HTTP: POST /index.html HTTP/1.1
[email protected]@.....n:..n..x.P.......(P.r.5h..POST /index.html HTTP/1.1
User-Agent: curl/7.29.0
Host: SERVER
Accept: */*
Content-Length: 1258
Expect: 100-continue
Content-Type: multipart/form-data; boundary=----------------------------61700007fd77

.........7.?`)+.
17:58:31.766389 IP SERVER.http > CLIENT.56184: Flags [.], ack 232, win 1817, length 0
E..(;.....Ks..n...n:.P.x...(....P.... ........................
17:58:32.692418 IP CLIENT.56184 > SERVER.http: Flags [P.], seq 232:486, ack 1, win 29200, length 254: HTTP
E..&[email protected]@.....n:..n..x.P.......(P.r.5...------------------------------61700007fd77
< Data for packet 2 >

..........8.?`..
17:58:32.856104 IP SERVER.http > CLIENT.56184: Flags [.], ack 486, win 1563, length 0
E..(;.....Km..n...n:.P.x...(....P.... ..............x...8.?`R.
17:58:32.856139 IP CLIENT.56184 > SERVER.http: Flags [P.], seq 486:1490, ack 1, win 29200, length 1004: HTTP
[email protected]@.....n:..n..x.P.......(P.r.8m..[ID]
< Data for packet 3 >

....8.?`...6....
17:58:32.919921 IP SERVER.http > CLIENT.56184: Flags [.], ack 1490, win 2048, length 0
E..(;.....Kl..n...n:.P.x...(....P....O..................8.?`O.
17:58:32.924255 IP SERVER.http > CLIENT.56184: Flags [P.], seq 1:121, ack 1490, win 2048, length 120: HTTP: HTTP/1.0 200 OK
E...;.....J...n...n:.P.x...(....P....o..HTTP/1.0 200 OK
Content-Type: text/javascript
Access-Control-Allow-Origin: *
Content-length: 0
Connection: close

........8.?`._.7

It’s very clean: as I read this, we send the first packet, it is acknowledge, we send the second, etc., and eventually we close the connection after receiving a nice happy response.

However, the communication from the requests module is a lot dirtier:

18:24:46.311756 IP CLIENT.56212 > SERVER.http: Flags [P.], seq 1:289, ack 1, win 29200, length 288: HTTP: POST /index.html HTTP/1.1
[email protected]@.....n:..n....P.9.N..v.P.r.5...POST /index.html HTTP/1.1
Host: SERVER
User-Agent: test client
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive
Content-Length: 1247
Content-Type: multipart/form-data; boundary=d8a887dda41b5a35f61ccf79b26d7b4e

........^.?`.C..
18:24:46.311772 IP CLIENT.56212 > SERVER.http: Flags [.], seq 289:1313, ack 1, win 29200, length 1024: HTTP
E..([email protected]@.....n:..n....P.9.n..v.P.r.8...--d8a887dda41b5a35f61ccf79b26d7b4e
< Data from packet 2 >

........^.?`+Z..
18:24:46.311777 IP CLIENT.56212 > SERVER.http: Flags [P.], seq 1313:1536, ack 1, win 29200, length 223: HTTP
[email protected]@.....n:..n....P.9.n..v.P.r.5`..
< Data from packet 3 >

................
18:24:46.525743 IP SERVER.http > CLIENT.56212: Flags [.], ack 289, win 1760, length 0
E..([D....,%..n...n:.P....v..9.nP....0..................^.?`..
18:24:46.800583 IP CLIENT.56212 > SERVER.http: Flags [.], seq 289:1313, ack 1, win 29200, length 1024: HTTP
E..([email protected]@.....n:..n....P.9.n..v.P.r.8...--d8a887dda41b5a35f61ccf79b26d7b4e
< Data from packet 2, again >

........^.?`.../
18:24:46.803014 IP SERVER.http > CLIENT.56212: Flags [.], ack 1313, win 2048, length 0
E..([E....,$..n...n:.P....v..9.nP...................p...^.?`.R
18:24:46.803033 IP CLIENT.56212 > SERVER.http: Flags [P.], seq 1313:1536, ack 1, win 29200, length 223: HTTP
[email protected]@.....n:..n....P.9.n..v.P.r.5`..
< Data from packet 3, again >

.........^.?`k?.
18:24:46.813645 IP SERVER.http > CLIENT.56212: Flags [F.], seq 1, ack 1536, win 1825, length 0
E..([F....,#..n...n:.P....v..9.MP..!....................^.?`h.
18:24:46.813813 IP CLIENT.56212 > SERVER.http: Flags [F.], seq 1536, ack 2, win 29200, length 0
E..([email protected]@.....n:..n....P.9.M..v.P.r.4...........^.?`...0
18:24:46.814339 IP SERVER.http > CLIENT.56212: Flags [.], ack 1537, win 1824, length 0
E..([G....,"..n...n:.P....v..9.NP.. ....................^.?`..
18:24:46.816550 IP CLIENT.56214 > SERVER.http: Flags [S], seq 1228421461, win 29200, options [mss 1460,sackOK,TS val 3666736130 ecr 0,nop,wscale 7], length 0
E..<[email protected]@.8...n:..n....PI89U......r.4..........
................^.?`0..0....
18:24:46.817006 IP SERVER.http > CLIENT.56214: Flags [S.], seq 416609351, ack 1228421462, win 2048, options [mss 1460], length 0
E..,[H....,...n...n:.P.....GI89V`.......................^.?`..
18:24:46.817021 IP CLIENT.56214 > SERVER.http: Flags [.], ack 1, win 29200, length 0
E..([email protected]@.9...n:..n....PI89V...HP.r.4...........^.?`.0.0
18:24:46.817049 IP CLIENT.56214 > SERVER.http: Flags [P.], seq 1:289, ack 1, win 29200, length 288: HTTP: POST /index.html HTTP/1.1
[email protected]@.7...n:..n....PI89V...HP.r.5...POST /index.html HTTP/1.1
Host: SERVER
User-Agent: test (EPICS base 7.0.4-E3-7.0.4-patch IOC)
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive
Content-Length: 1247
Content-Type: multipart/form-data; boundary=04a493e5def4d0baf76026663f63ae61

........^.?`.g.0
18:24:46.817063 IP CLIENT.56214 > SERVER.http: Flags [.], seq 289:1313, ack 1, win 29200, length 1024: HTTP
E..([email protected]@.5...n:..n....PI8:v...HP.r.8...--04a493e5def4d0baf76026663f63ae61
< Data from packet 2, again! >

....p...^.?`.z.0
18:24:46.817068 IP CLIENT.56214 > SERVER.http: Flags [P.], seq 1313:1536, ack 1, win 29200, length 223: HTTP
E....[@[email protected]/..n:..n....PI8>v...HP.r.5`..
< Data from packet 3, again! >

etc.

The first things that I note are that in this case, all three packets are sent before the first one is acknowledged; after that the second packet is sent, acknowledged, then the third packet is sent.

However, after this, the whole thing is sent again for some reason, and we never get an HTTP/1.0 200 OK message together with a good response.

I know that the HTTP headers that are being sent between the two are slightly different, but even synchronising those does not fix the communication betwee the two. I also note that the packet size is different, but I cannot imagine that being an issue.

I also note that the packets sent via curl all have the PUSH flag set, but this is done inconsistently on the python side. But other than that, I don’t really see a difference.

So my question is: Why are the two acting differently, and how can I get the python requests module to act more like curl in this case?

Source: Python Questions

LEAVE A COMMENT