Re: Proposal: Compression of large packets

["Allan Latham" <alatham,AT,flexsys-group,DOT,com>]

I added this to an earlier version along with compression of the
encapsulated tcp and ip headers. It still works well but not available on
2.4 kernels (no problems - our routers run on 2.2). I have posted the source
to the group so I expect you can find it in the archives. At least it gives
you a start.

Best regards

Allan

----- Original Message -----
From: "Josef Drexler" <jdrexler,AT,josefsbox,DOT,cjb,DOT,net>
To: "CIPE Mailing List" <cipe-l,AT,inka,DOT,de>
Sent: Friday, November 30, 2001 7:45 PM
Subject: Proposal: Compression of large packets

>
> Previously I used to have a VPN tunneled by ssh, with all the problems
> that made Olaf Titz develop CIPE.  I like CIPE a lot, but the one
> advantage that ssh had was it could compress the data stream.
>
> I understand that doing this based on datagrams is more difficult, however
> I did some statistics on my connection, using tcpdump and a perl script
> that did the compression from tcpdumped packets to see how much they could
> have been compressed.  The results from about a day of traffic are at the
> end of this email.
>
> What I found is that compression is only efficient for very large packets
> (no surprise there);  basically only packets close to the MTU had a
> significant compression ratio.  However, these make up for more than 50%
> of the traffic, while only being less than 10% of the packets.  My tests
> showed that compressing these gives a ratio of around 50-60%, good enough
> to improve the bandwidth significantly.
>
> Of course this would require a relatively fast computer, but that's true
> for compressing with ssh as well.  However, by selecting a minimum packet
> size for which compression is attempted, the performance penalty would
> only occur for large packets, and for these it might well be an increase
> in performance, depending on the bandwidth (i. e., is it faster to
> transfer an uncompressed packet, or to compress and transfer the
> compressed packet).
>
> It would also be useful to make compression asymmetric, for example for
> ADSL where only one direction is so slow that it would benefit from
> compression.
>
>
> CONCLUSION
>
> Compression of large packets can significantly improve bandwidth while
> keeping latency low for small packets.
>
>
> WHAT NEEDS TO BE DONE
>
> - decide what compression scheme to use
> - make a new option to enable compression and choose minimum packet size
> - add a protocol to negotaite compression with peer (probably in a
>   CT_CONFREQ or CT_CONF
> - define a bit in P to indicate whether a packet is compressed
> - do the actual compression and decompression
> - anything else?
>
>
> STATISTICAL RESULTS
>
> Legend:
> Size Size of the packets in bytes (octets); up to the next entry
> Number How many packets were transferred with this size
> Pct Percentage of packets with this size
> PBytes  Percentage of total bytes to all traffic
> AvgComp Average compression ratio using Zlib's "compress" function, in
percent
> PctComp Percentage of packets that could successfully be compressed
> TotalB  Total bytes transferred
> BSaved  Bytes saved through compression (only where successful)
> PSaved  Percentage of bytes saved to all traffic
>
> Size    Number  Pct     PBytes  AvgComp PctComp TotalB  BSaved  PSaved
> 0       4991     37      11     118       0     255181  0        11
> 64      6503     49      22     113       0     501071  31       22
> 128     337       2       2     106       1     50481   21        2
> 192     106       0       1     104       1     22649   5         1
> 256     101       0       1      99      15     28552   1089      1
> 320     77        0       1     100      15     26930   521       1
> 384     72        0       1      91      55     29328   2921      1
> 448     32        0       0      98      18     15151   549       0
> 512     12        0       0      99      16     6516    143       0
> 576     8         0       0      75      62     4869    1217      0
> 640     23        0       0      81      52     15325   3005      0
> 704     11        0       0      97       9     8136    303       0
> 768     14        0       0      93      21     11310   857       0
> 832     11        0       0      98       9     9357    273       0
> 896     7         0       0      90      28     6532    695       0
> 960     6         0       0      90      16     5895    586       0
> 1024    14        0       0      98       7     14681   423       0
> 1088    10        0       0      91      20     11275   1028      0
> 1152    5         0       0      74      60     5922    1540      0
> 1216    14        0       0      82      42     17398   3157      0
> 1280    6         0       0      79      50     7867    1679      0
> 1344    10        0       0     100      10     13818   11        0
> 1408    819       6      52      57      90     1171427 504134   29
>
>
> --
>    Josef Drexler                 |    http://publish.uwo.ca/~jdrexler/
> ---------------------------------+---------------------------------------
>  Please help Conserve Gravity    | Email address is *valid*.
>  Carry a helium balloon.         | Don't remove the "nospam" part.
>
>
> --
> Message sent by the cipe-l,AT,inka,DOT,de mailing list.
> Unsubscribe: mail majordomo,AT,inka,DOT,de, "unsubscribe cipe-l" in body
> Other commands available with "help" in body to the same address.
> CIPE info and list archive:
<URL:http://sites.inka.de/~bigred/devel/cipe.html>
>