Steam Compression - Forum

johnm60

2010-06-30, 16:40 #1

Subject: Steam Compression

Hi there

I have seen other threads that talk around this subject, but could you clarify how this works with Matrix.

I have been testing Matrix against Isode M-Link. The XMPP client has a compression property and I have tried setting this to both false and true and cannot see a difference in the conversation between the client and server:

With compression=False;

[4828] xmppClient_OnSendXml: <stream:stream xmlns:stream="http://etherx.jabber.org/streams" xmlns="jabber:client" to="acme.com" version="1.0" >
[4828] xmppClient_OnReceiveXml: <stream:stream xmlns:stream="http://etherx.jabber.org/streams" xmlns="jabber:client" from="acme.com" id="0bc3d22667c31837" version="1.0" >
[4828] xmppClient_OnReceiveXml: <stream:features xmlns:stream="http://etherx.jabber.org/streams">
[4828]   <mechanisms xmlns="urn:ietf:params:xml:ns:xmpp-sasl">
[4828]    <mechanism>SCRAM-SHA-1</mechanism>
[4828]    <mechanism>PLAIN</mechanism>
[4828]    <mechanism>NTLM</mechanism>
[4828]    <mechanism>GSSAPI</mechanism>
[4828]    <mechanism>GSS-SPNEGO</mechanism>
[4828]    <mechanism>DIGEST-MD5</mechanism>
[4828]    <mechanism>CRAM-MD5</mechanism>
[4828]   </mechanisms>
[4828]   <compression xmlns="http://jabber.org/features/compress">
[4828]    <method>zlib</method>
[4828]   </compression>
[4828]   <ver xmlns="urn:xmpp:features:rosterver">
[4828]    <optional />
[4828]   </ver>
[4828] </stream:features>
[4828] xmppClient_OnSendXml: <auth mechanism="DIGEST-MD5" xmlns="urn:ietf:params:xml:ns:xmpp-sasl" />
[4828] xmppClient_OnReceiveXml: <challenge xmlns="urn:ietf:params:xml:ns:xmpp-sasl">bm9uY2U9IlRFcDJWRnBLQ1NVQlluOUtKendSUVJkR01FRldYRnBPRDNwZEJ4MC9FWHc9IixyZWFsbT0iYWNtZS5jb20iLHFvcD0iYXV0aCIsbWF4YnVmPTE2Mzg0LGNoYXJzZXQ9dXRmLTgsYWxnb3JpdGhtPW1kNS1zZXNz</challenge>
[4828] xmppClient_OnSendXml: <response xmlns="urn:ietf:params:xml:ns:xmpp-sasl">dXNlcm5hbWU9ImpvbGx5LmpvaG4iLHJlYWxtPSJhY21lLmNvbSIsbm9uY2U9IlRFcDJWRnBLQ1NVQlluOUtKendSUVJkR01FRldYRnBPRDNwZEJ4MC9FWHc9Iixjbm9uY2U9IjViNmMzMTM3M2VmODk4NmIxYWVmODU5OTExYzljMzdmYzBjYzFjYWIzMTU0NjRmOTcxNGYyZDc4NTViZTI5MWMiLG5jPTAwMDAwMDAxLHFvcD1hdXRoLGRpZ2VzdC11cmk9InhtcHAvYWNtZS5jb20iLGNoYXJzZXQ9dXRmLTgscmVzcG9uc2U9MzFlZDhmMmNkNThjNzZjMDlkYjVlMjM4YTRlNzllMzI=</response>
[4828] xmppClient_OnReceiveXml: <success xmlns="urn:ietf:params:xml:ns:xmpp-sasl">cnNwYXV0aD1hYzhmMzlkYWY4MzBkMzdhM2E3Y2Y2MjMyZmI5YzkwNw==</success>
[4828] xmppClient_OnSendXml: <stream:stream xmlns:stream="http://etherx.jabber.org/streams" xmlns="jabber:client" to="acme.com" version="1.0" >
[4828] xmppClient_OnReceiveXml: <stream:stream xmlns:stream="http://etherx.jabber.org/streams" xmlns="jabber:client" from="acme.com" id="c10d1c22c34e4bad" version="1.0" >
[4828] xmppClient_OnReceiveXml: <stream:features xmlns:stream="http://etherx.jabber.org/streams">
[4828]   <bind xmlns="urn:ietf:params:xml:ns:xmpp-bind">
[4828]    <required />
[4828]   </bind>
[4828]   <session xmlns="urn:ietf:params:xml:ns:xmpp-session" />
[4828]   <compression xmlns="http://jabber.org/features/compress">
[4828]    <method>zlib</method>
[4828]   </compression>
[4828]   <ver xmlns="urn:xmpp:features:rosterver">
[4828]    <optional />
[4828]   </ver>
[4828] </stream:features>
[4828] xmppClient_OnSendXml: <iq id="MX_1" type="set" xmlns="jabber:client">
[4828]   <bind xmlns="urn:ietf:params:xml:ns:xmpp-bind">
[4828]    <resource>MatriX</resource>
[4828]   </bind>
[4828] </iq>
[4828] xmppClient_OnReceiveXml: <iq to="jolly.john@acme.com/MatriX" type="result" id="MX_1" xmlns="jabber:client">
[4828]   <bind xmlns="urn:ietf:params:xml:ns:xmpp-bind">
[4828]    <jid>jolly.john@acme.com/MatriX</jid>
[4828]   </bind>
[4828] </iq>
[4828] xmppClient_OnSendXml: <iq id="MX_2" type="set" xmlns="jabber:client">
[4828]   <session xmlns="urn:ietf:params:xml:ns:xmpp-session" />
[4828] </iq>
[4828] xmppClient_OnReceiveXml: <iq from="acme.com" type="result" id="MX_2" xmlns="jabber:client" />

and with Compression=True;

[4332] xmppClient_OnSendXml: <stream:stream xmlns:stream="http://etherx.jabber.org/streams" xmlns="jabber:client" to="acme.com" version="1.0" >
[4332] xmppClient_OnReceiveXml: <stream:stream xmlns:stream="http://etherx.jabber.org/streams" xmlns="jabber:client" from="acme.com" id="d19cc9c961fbe8b4" version="1.0" >
[4332] xmppClient_OnReceiveXml: <stream:features xmlns:stream="http://etherx.jabber.org/streams">
[4332]   <mechanisms xmlns="urn:ietf:params:xml:ns:xmpp-sasl">
[4332]    <mechanism>SCRAM-SHA-1</mechanism>
[4332]    <mechanism>PLAIN</mechanism>
[4332]    <mechanism>NTLM</mechanism>
[4332]    <mechanism>GSSAPI</mechanism>
[4332]    <mechanism>GSS-SPNEGO</mechanism>
[4332]    <mechanism>DIGEST-MD5</mechanism>
[4332]    <mechanism>CRAM-MD5</mechanism>
[4332]   </mechanisms>
[4332]   <compression xmlns="http://jabber.org/features/compress">
[4332]    <method>zlib</method>
[4332]   </compression>
[4332]   <ver xmlns="urn:xmpp:features:rosterver">
[4332]    <optional />
[4332]   </ver>
[4332] </stream:features>
[4332] xmppClient_OnSendXml: <auth mechanism="DIGEST-MD5" xmlns="urn:ietf:params:xml:ns:xmpp-sasl" />
[4332] xmppClient_OnReceiveXml: <challenge xmlns="urn:ietf:params:xml:ns:xmpp-sasl">bm9uY2U9Ik9tUk9ZMDVySkg5dEt4UU5lMzFLRGdncU1qZGVaa3h3U0hRYUhSWTRLUnM9IixyZWFsbT0iYWNtZS5jb20iLHFvcD0iYXV0aCIsbWF4YnVmPTE2Mzg0LGNoYXJzZXQ9dXRmLTgsYWxnb3JpdGhtPW1kNS1zZXNz</challenge>
[4332] xmppClient_OnSendXml: <response xmlns="urn:ietf:params:xml:ns:xmpp-sasl">dXNlcm5hbWU9ImpvbGx5LmpvaG4iLHJlYWxtPSJhY21lLmNvbSIsbm9uY2U9Ik9tUk9ZMDVySkg5dEt4UU5lMzFLRGdncU1qZGVaa3h3U0hRYUhSWTRLUnM9Iixjbm9uY2U9ImNiYjBmMjZkNmYxMTcyYTk3MDQ2NjAwMTQzZGZkYzk4NjRiMTMzZDhjOGZiYzE0NDdlZTExNTQ3NDU1YTg2ZjUiLG5jPTAwMDAwMDAxLHFvcD1hdXRoLGRpZ2VzdC11cmk9InhtcHAvYWNtZS5jb20iLGNoYXJzZXQ9dXRmLTgscmVzcG9uc2U9OGQwODM2NjNkODBhOTQ4MmVkODliY2VlNGU5NTExYjk=</response>
[4332] xmppClient_OnReceiveXml: <success xmlns="urn:ietf:params:xml:ns:xmpp-sasl">cnNwYXV0aD0yOTFlNjZmNjFiNzM2MDZjNjFhOWIyYzU5NWJiYmNmMA==</success>
[4332] xmppClient_OnSendXml: <stream:stream xmlns:stream="http://etherx.jabber.org/streams" xmlns="jabber:client" to="acme.com" version="1.0" >
[4332] xmppClient_OnReceiveXml: <stream:stream xmlns:stream="http://etherx.jabber.org/streams" xmlns="jabber:client" from="acme.com" id="ff0ed970ada5b16c" version="1.0" >
[4332] xmppClient_OnReceiveXml: <stream:features xmlns:stream="http://etherx.jabber.org/streams">
[4332]   <bind xmlns="urn:ietf:params:xml:ns:xmpp-bind">
[4332]    <required />
[4332]   </bind>
[4332]   <session xmlns="urn:ietf:params:xml:ns:xmpp-session" />
[4332]   <compression xmlns="http://jabber.org/features/compress">
[4332]    <method>zlib</method>
[4332]   </compression>
[4332]   <ver xmlns="urn:xmpp:features:rosterver">
[4332]    <optional />
[4332]   </ver>
[4332] </stream:features>
[4332] xmppClient_OnSendXml: <iq id="MX_1" type="set" xmlns="jabber:client">
[4332]   <bind xmlns="urn:ietf:params:xml:ns:xmpp-bind">
[4332]    <resource>MatriX</resource>
[4332]   </bind>
[4332] </iq>
[4332] xmppClient_OnReceiveXml: <iq to="jolly.john@acme.com/MatriX" type="result" id="MX_1" xmlns="jabber:client">
[4332]   <bind xmlns="urn:ietf:params:xml:ns:xmpp-bind">
[4332]    <jid>jolly.john@acme.com/MatriX</jid>
[4332]   </bind>
[4332] </iq>
[4332] xmppClient_OnSendXml: <iq id="MX_2" type="set" xmlns="jabber:client">
[4332]   <session xmlns="urn:ietf:params:xml:ns:xmpp-session" />
[4332] </iq>
[4332] xmppClient_OnReceiveXml: <iq from="acme.com" type="result" id="MX_2" xmlns="jabber:client" />

As you can see, the server offers zlib on both occasions.

Am I meant to see a different negotiation? The reason I am asking is that I have been performance testing with large (1000 user) rosters and see no noticeable performance improvement with compression set to true.

Thanks
John

Alex

2010-06-30, 17:29 #2

the problem is that Mlink is using another namespace there for compression:
http://jabber.org/features/compress

All other servers I tested in the past used:
http://jabber.org/protocol/compress

I will check the docs and come back to you. A fix to support both namespaces would be pretty easy.

Alex

Alex

2010-06-30, 19:22 #3

uploaded a new version where stream compression works with MLink servers now.
When you are on TlS you should disable compression because TLS has build in compression.

Alex

johnm60

2010-06-30, 19:33 #4

Thanks Alex

Just tried that and got

[6740] xmppClient_OnSendXml: <stream:stream xmlns:stream="http://etherx.jabber.org/streams" xmlns="jabber:client" to="acme.com" version="1.0" >
[6740] xmppClient_OnReceiveXml: <stream:stream xmlns:stream="http://etherx.jabber.org/streams" xmlns="jabber:client" from="acme.com" id="06c84f526f2138f5" version="1.0" >
[6740] xmppClient_OnReceiveXml: <stream:features xmlns:stream="http://etherx.jabber.org/streams">
[6740]   <mechanisms xmlns="urn:ietf:params:xml:ns:xmpp-sasl">
[6740]    <mechanism>SCRAM-SHA-1</mechanism>
[6740]    <mechanism>PLAIN</mechanism>
[6740]    <mechanism>NTLM</mechanism>
[6740]    <mechanism>GSSAPI</mechanism>
[6740]    <mechanism>GSS-SPNEGO</mechanism>
[6740]    <mechanism>DIGEST-MD5</mechanism>
[6740]    <mechanism>CRAM-MD5</mechanism>
[6740]   </mechanisms>
[6740]   <compression xmlns="http://jabber.org/features/compress">
[6740]    <method>zlib</method>
[6740]   </compression>
[6740]   <ver xmlns="urn:xmpp:features:rosterver">
[6740]    <optional />
[6740]   </ver>
[6740] </stream:features>
[6740] xmppClient_OnSendXml: <auth mechanism="DIGEST-MD5" xmlns="urn:ietf:params:xml:ns:xmpp-sasl" />
[6740] xmppClient_OnReceiveXml: <challenge xmlns="urn:ietf:params:xml:ns:xmpp-sasl">bm9uY2U9Ik1HVVlJd2t4S2pjSU9qOEpaQkpUYURVY2JXRXFDUWNkUVRWS1AzVTJHMGc9IixyZWFsbT0iYWNtZS5jb20iLHFvcD0iYXV0aCIsbWF4YnVmPTE2Mzg0LGNoYXJzZXQ9dXRmLTgsYWxnb3JpdGhtPW1kNS1zZXNz</challenge>
[6740] xmppClient_OnSendXml: <response xmlns="urn:ietf:params:xml:ns:xmpp-sasl">dXNlcm5hbWU9ImpvbGx5LmpvaG4iLHJlYWxtPSJhY21lLmNvbSIsbm9uY2U9Ik1HVVlJd2t4S2pjSU9qOEpaQkpUYURVY2JXRXFDUWNkUVRWS1AzVTJHMGc9Iixjbm9uY2U9IjFhNjlkY2YzMGNmYzhlOWU3YTMxZmRlNTQzNWI2Njg1ODhlOTA5MzM3NzQyMTZlNDdiNDdhODFhM2RmM2YzMmIiLG5jPTAwMDAwMDAxLHFvcD1hdXRoLGRpZ2VzdC11cmk9InhtcHAvYWNtZS5jb20iLGNoYXJzZXQ9dXRmLTgscmVzcG9uc2U9MDVlYmZhZjU2NTc4ODY5NWQ0YTNjZGExZGEzNDA2NTk=</response>
[6740] xmppClient_OnReceiveXml: <success xmlns="urn:ietf:params:xml:ns:xmpp-sasl">cnNwYXV0aD1kZjU2MDk3N2Q2NDNkMDdjYThiM2VhYjQ2MTYyZDJhOQ==</success>
[6740] xmppClient_OnSendXml: <stream:stream xmlns:stream="http://etherx.jabber.org/streams" xmlns="jabber:client" to="acme.com" version="1.0" >
[6740] xmppClient_OnReceiveXml: <stream:stream xmlns:stream="http://etherx.jabber.org/streams" xmlns="jabber:client" from="acme.com" id="cfc927869075803d" version="1.0" >
[6740] xmppClient_OnReceiveXml: <stream:features xmlns:stream="http://etherx.jabber.org/streams">
[6740]   <bind xmlns="urn:ietf:params:xml:ns:xmpp-bind">
[6740]    <required />
[6740]   </bind>
[6740]   <session xmlns="urn:ietf:params:xml:ns:xmpp-session" />
[6740]   <compression xmlns="http://jabber.org/features/compress">
[6740]    <method>zlib</method>
[6740]   </compression>
[6740]   <ver xmlns="urn:xmpp:features:rosterver">
[6740]    <optional />
[6740]   </ver>
[6740] </stream:features>
[6740] xmppClient_OnSendXml: <compress xmlns="http://jabber.org/protocol/compress">
[6740]   <method>zlib</method>
[6740] </compress>
[6740] xmppClient_OnReceiveXml: <compressed xmlns="http://jabber.org/protocol/compress" />

So the <compressed> stanza now means that the who stream from now on is compressed?

John

Alex

2010-06-30, 19:36 #5

yes, after the compressed stanza the stream is compressed. The log handlers give you the uncompressed results. When you use a socket sniffer you can see that after this stanza the socket data is compressed and not human readable anymore.

Alex

johnm60

2010-06-30, 20:08 #6

Thanks

Sorry to have kept you so busy today :-)

John

Alex

2010-07-01, 13:15 #7

no problem. You're welcome.

What Mlink does is correct. The specs were updated a whole ago. This the problem when implementing extensions which are not final and can change. So thanks to you for finding this problem.

Alex