OnClose event occur frequently on BOSH connection

Yasu

2012-02-08, 02:30 #1

Member since Sep 2011 · 22 posts
Group memberships: Members

Show profile ·

Link to this post

Subject: OnClose event occur frequently on BOSH connection

In my network environment, System.Net.WebException occur frequently.
I have a trouble that the session closes so frequently on XMPP over BOSH connection.(XMPP(not over BOSH) has no problem)

When Matrix detect System.Net.WebException once, Matrix closes the session?
Debug log shows that "</stream:stream>" is sent from Matrix after receiving OnErrorEvent with System.Net.WebException.

I sent ping just after receiving OnClose event, but WebException was not occurred.

I think It is possible to avoid this kind of session closing, by retrying WebRequest some times.

I want to avoid session close as much as possible, do you have some idea ?

Matrix:version 1.4.4.3
Server:Openfire

Alex

2012-02-08, 07:50 #2

Member since Feb 2003 · 4449 posts · Location: Germany
Group memberships: Administrators, Members

Show profile ·

Link to this post

Quote by Yasu:
In my network environment, System.Net.WebException occur frequently.
I have a trouble that the session closes so frequently on XMPP over BOSH connection.(XMPP(not over BOSH) has no problem)

Openfire closed Bosh sessions when there was no traffic for a while. Thats not nice.

Quote by Yasu:
When Matrix detect System.Net.WebException once, Matrix closes the session?
Debug log shows that "</stream:stream>" is sent from Matrix after receiving OnErrorEvent with System.Net.WebException.

I sent ping just after receiving OnClose event, but WebException was not occurred.

When you get an OnClose then the connection is closed, you cannot and should not send anything after the OnClose.

Quote by Yasu:
I think It is possible to avoid this kind of session closing, by retrying WebRequest some times.

I want to avoid session close as much as possible, do you have some idea ?
Matrix:version 1.4.4.3
Server:Openfire

I have attached a Beta of 1.4.4.4.
This version has a completly redesigned BOSH component and should send keep alives on BOSH to keep your session on Openfire alive.
Can you please test this version and let me know if it helps?

Alex

This post was edited on 2012-02-09, 08:00 by Alex.

Yasu

2012-02-09, 07:55 #3

Member since Sep 2011 · 22 posts
Group memberships: Members

Show profile ·

Link to this post

Thanks for beta version.

I'm testing now, It seems woking fine for us!!
I'll let you know when I finish testing.

I found a problem this version.
sometimes, suddenly I receive OnClose event, without receiving OnError event.

This is part of log.

OnReceivedBody: <body xmlns="http://jabber.org/protocol/httpbind" type="terminate" condition="policy-violation"></body>
sender thread continue
sender thread ended
Cancel request A
DestroyKeepAliveTimer
Cancel request B
BoshSocket:OnError: can be recovered
Last Send A:16:41:14
Last Send B:16:41:13
Last Send =>16:41:14
InactiveLeft: 30
Last Send A:16:41:14
Last Send B:16:41:13
Last Send =>16:41:14
Sleep: 5000
RECV:</stream:stream>
SEND:</stream:stream>
OnClose
Work Done A ==> Pulse

This post was edited on 2012-02-09, 07:57 by Yasu.

Alex

2012-02-09, 08:00 #4

Member since Feb 2003 · 4449 posts · Location: Germany
Group memberships: Administrators, Members

Show profile ·

Link to this post

Quote by Yasu:
sometimes, suddenly I receive OnClose event, without receiving OnError event.

This is part of log.

OnReceivedBody: <body xmlns="http://jabber.org/protocol/httpbind" type="terminate" condition="policy-violation"></body>
sender thread continue
sender thread ended
Cancel request A
DestroyKeepAliveTimer
Cancel request B
BoshSocket:OnError: can be recovered
Last Send A:16:41:14
Last Send B:16:41:13
Last Send =>16:41:14
InactiveLeft: 30
Last Send A:16:41:14
Last Send B:16:41:13
Last Send =>16:41:14
Sleep: 5000
RECV:</stream:stream>
SEND:</stream:stream>
OnClose
Work Done A ==> Pulse

Your log shows an OnError event.
BoshSocket:OnError: can be recovered

Alex

Alex

2012-02-09, 08:02 #5

Member since Feb 2003 · 4449 posts · Location: Germany
Group memberships: Administrators, Members

Show profile ·

Link to this post

There is also an error condition in youir log:

do you have any idea which policy you are violating?

Alex

Yasu

2012-02-09, 08:31 #6

Member since Sep 2011 · 22 posts
Group memberships: Members

Show profile ·

Link to this post

Your log shows an OnError event.
BoshSocket:OnError: can be recovered

My log shows OnError, but in this case, XmppClient's OnError was not called from MatriX.

<body xmlns="http://jabber.org/protocol/httpbind" type="terminate" condition="policy-violation"></body>

do you have any idea which policy you are violating?

hmmm.. I have no idea...
but the last sending message before receiving this violation to the XMPP server was blank message,
Are you sending blank message?

Alex

2012-02-09, 08:35 #7

Member since Feb 2003 · 4449 posts · Location: Germany
Group memberships: Administrators, Members

Show profile ·

Link to this post

Quote by Yasu:
hmmm.. I have no idea...
but the last sending message before receiving this violation to the XMPP server was blank message,
Are you sending blank message?

yes, the new BOSH code support also keep alives now because Openfire still has a problem (bug?) and kills idle connections on BOSH.
So when the keep alive interval is elapsed is sends a empty <body/> element.
The policy-violation is currently not handled in the code. When it receives this violation it should immediately Close the session. I put this on my TODO list for the next release.

Yasu

2012-02-10, 08:17 #8

Member since Sep 2011 · 22 posts
Group memberships: Members

Show profile ·

Link to this post

The policy-violation is currently not handled in the code. When it receives this violation it should immediately Close the session. I put this on my TODO list for the next release.

I see.
It'll be better for us that it says OnError event before Close the settion.

And my test was done, the beta of 1.4.4.4 works fine.

Thanks.

Yasu

2012-02-16, 05:08 #9

Member since Sep 2011 · 22 posts
Group memberships: Members

Show profile ·

Link to this post

About that policy-violation, perhaps this is the openfire's bug.
http://issues.igniterealtime.org/browse/OF-…?page=com.at…

Alex

2012-02-16, 07:45 #10

Member since Feb 2003 · 4449 posts · Location: Germany
Group memberships: Administrators, Members

Show profile ·

Link to this post

I have posted in the Ingiterealtime forums about this problem. But I got no reply yet. As long as I don't know why the server sends you this error, and if its a server bug or not I can't really help you.

Alex

Yasu

2012-02-17, 10:33 #11

Member since Sep 2011 · 22 posts
Group memberships: Members

Show profile ·

Link to this post

Thanks for your help a lot!

I reanalyzed this problem with server side guy's help.
And we figured out the reason why the WebException(404 not found) was occured.

It seems that empty body HTTP request is faster than polling interval in specific condition.
I attached the details image, and I'll send you a packet data, please look these one.

Matrix:version 1.4.4.3 ( not using 1.4.4.4 beta version )
Server:Openfire

Thanks.

The author has attached one file to this post:

policy-violation.pdf 33.7 kBytes
You have no permission to open this file.

Alex

2012-02-17, 10:50 #12

Member since Feb 2003 · 4449 posts · Location: Germany
Group memberships: Administrators, Members

Show profile ·

Link to this post

MatriX sends only empty body requests only when the keep alive interval is elapsed, or when there is no open web request and there is no pending data to send.
We use BOSH with 2 connections, this is not polling. So the polling value shouldn't matter at all.
see: http://xmpp.org/extensions/xep-0124.html

Do you have a HTTP proxy between MatriX and your Openfre server? We have seen cases where the HTTP proxies make lots of problems. If this is your case then try directly without HTTP proxy.

Alex

Alex

2012-02-17, 10:57 #13

Member since Feb 2003 · 4449 posts · Location: Germany
Group memberships: Administrators, Members

Show profile ·

Link to this post

And please use the latest Beta I sent you. This has many BOSH improvements.

What you can try as a workaround is the following:
When MatriX creates a new Webrequest is always raises the OnCreateWebRequest. This event is meant for adding additional HTTP headers and similar stuff. You can try to put a Thread.Sleep(x) there. This will stop MatriX from sending the HTTP request too fast.
Let me know if this solves your problem.

Yasu

2012-02-18, 02:42 #14

Member since Sep 2011 · 22 posts
Group memberships: Members

Show profile ·

Link to this post

In reply to post #12

Sending empty request means 'keep alive' and 'polling' also.
I heard that the server is treating empty message as polling.

MatriX sends only empty body requests only when the keep alive interval is elapsed

#I'm sorry, if I was misunderstanding what you mentioned.

Connection Manager send to client an empty response to keep alive, before the keep alive interval('wait' parameter) is elapsed,
so client has no need to watch the keep alive interval.
When Connection Manager receive request from client, Connection Manager send you empty response to keep http session less than 2(2 is 'requests' parameter).

When client receive response, client should send empty request(keep alive) immediately if there is no open web request.
Furthermore client should not send empty request before the polling interval('polling' parameter) is elapsed.
If client break these specification, overactivity occur.

'polling' -- This attribute specifies the shortest allowable polling interval (in seconds). This enables the client to not send empty request elements more often than desired (see Polling Sessions and Overactivity)

'requests' -- This attribute enables the connection manager to limit the number of simultaneous requests the client makes (see Overactivity and Polling Sessions). The RECOMMENDED values are either "2" or one more than the value of the 'hold' attribute specified in the session request.

Do you have a HTTP proxy between MatriX and your Openfre server?

I'm sorry, We could not access Openfire without proxy because the server exist in out of our network.

And please use the latest Beta I sent you. This has many BOSH improvements.

What you can try as a workaround is the following:

Thanks for your suggestions!
I'll try the latest Beta version also, and add sleep to OnCreateWebRequest.

But if you were not concerning polling parameter, I think I can't avoid every overactivity because 5 second is too long to wait for every requests..
Can I detect the timing when empty request is sent ? If it's possible I think I can avoid overactivity.

Alex

2012-02-18, 11:56 #15

Member since Feb 2003 · 4449 posts · Location: Germany
Group memberships: Administrators, Members

Show profile ·

Link to this post

Quote by Yasu:
Sending empty request means 'keep alive' and 'polling' also.
I heard that the server is treating empty message as polling.

no, in order to be able to receive data at any time from the server the client needs always one open webrequest to the server.

Example:
1 request is open to the server and the server sends you message on this request. MatriX gets the message so 0 requests are open. But currently your clients has no stanzas to send to the server. So MatriX must open a new request with an empty body. This is how bosh works and this is no polling.

Quote by Yasu:
'polling' -- This attribute specifies the shortest allowable polling interval (in seconds). This enables the client to not send empty request elements more often than desired (see Polling Sessions and Overactivity)

When you use Bosh with only 1 Webrequest this is polling. MatriX is using Bosh with 2 request and does no polling.
MatriX sends polling=0 in the session request.

Quote by Yasu:
'requests' -- This attribute enables the connection manager to limit the number of simultaneous requests the client makes (see Overactivity and Polling Sessions). The RECOMMENDED values are either "2" or one more than the value of the 'hold' attribute specified in the session request.

MatriX send 2 here in the session request and does not create more than 2 requests at any time.

Quote by Yasu:
I'm sorry, We could not access Openfire without proxy because the server exist in out of our network.

You can install a local Openfire server on your developer machine.

Quote by Yasu:
But if you were not concerning polling parameter, I think I can't avoid every overactivity because 5 second is too long to wait for every requests..
Can I detect the timing when empty request is sent ? If it's possible I think I can avoid overactivity.

MatriX send 0 for this value. When your server ignores this then this is a server configuration issue or but. Or some problem with your proxy in the middle. We have lots of customers using BOSH under very high load, and we have never seen such problems.

Alex