Matrix.OnError - There is an unclosed literal string. Line 1, position 98305.

Lightbarrier

2016-09-30, 03:02 #1

Member since Jul 2014 · 52 posts
Group memberships: Members

Show profile ·

Link to this post

Subject: Matrix.OnError - There is an unclosed literal string. Line 1, position 98305.

Hi Alex,

Spec Details
MatriX Version: 1.6.0.3
.Net Version: 4.0
Connection Type: HTTP (I realize a socket connection is more reliable, but we have users that have to use http due to the server that they're connecting to. We do have cases where socket connections are ok, and we are using them. However, for some of our users it has to be http or nothing.)

We’re experiencing the following error below where users are being disconnected from the chat server without warning.

Error:
Matrix.OnError(object sender, ExceptionEventArgs error)
error.Exception.GetType(): Matrix.Net.BoshException
error.Exception.Message: BoshException
error.Exception.GetBaseException().GetType(): System.Xml.XmlException
error.Exception.GetBaseException().Message: There is an unclosed literal string. Line 1, position 98305.
Stack Trace of Matrix.OnError: In File

In the past I've seen similar errors when the message payload is to big with an example being the roster. This was usually resolved upon shortening the amount of data that the server is sending to the client. However, in this case the last message that the server is sending to the client is the presence message shown below which in no way matches the size given in the error message.

<body xmlns="http://jabber.org/protocol/httpbind" ack="2095285817">
  <presence xmlns="jabber:client" to="admin.mcojw-agent@mcojwvmgw/MatriX" from="mcorz@conference.mcojwvmgw/admin.mcojw">
    <show>chat</show>
    <x xmlns="http://jabber.org/protocol/muc#user">
      <item jid="admin.mcojw-agent@mcojwvmgw/MatriX" affiliation="owner" role="moderator" />
    </x>
  </presence>
</body>

From what I can see some exception is occurring within MatriX that I can't pinpoint. We've tried looking at Wireshark to see the packets coming to the client in case a message was received that never reached the 'OnReceiveXml' event however, it showed that the presence message shown above is indeed the last message sent from the Chat Server. Looking at the stack trace on the 'OnError' event (shown in the attachment file "Chat Trace Client - 9_27_18_5_41.txt" at the bottom of the page) I'm wondering if 'HttpWebRequest' has something to do with the error which I know you've talked about in other comments.

Have you seen a problem like this before? We could try updating to a latter version of MatriX however, we're wondering if you've discovered this issue and whither it has been resolved in a latter version (If possible we would like to stick with .Net 4.0 to avoid introducing any new issues in to our software). Please let us know if there's anything else we can give you that may help in understanding what the problem is.

Thanks for your help.

The author has attached one file to this post:

ChatTrace.zip 86 kBytes
You have no permission to open this file.

Alex

2016-09-30, 19:47 #2

Member since Feb 2003 · 4449 posts · Location: Germany
Group memberships: Administrators, Members

Show profile ·

Link to this post

yo are using a pretty old version of MatriX. I would suggest to update.

Is it possible that there is a syntax problem in the Xml?

or your network connection get lost and MatriX received only partial XML which causes the Xml parse exception?

The Xml logs provided get written after MatriX parsed the Xml. We would need the raw data for more diagnosis from a tool like Wireshark or Fiddler.

Lightbarrier

2016-10-01, 02:34 #3

Member since Jul 2014 · 52 posts
Group memberships: Members

Show profile ·

Link to this post

Quote by Alex:
yo are using a pretty old version of MatriX. I would suggest to update.

We tried updating to the latest version of MatriX Version 2.0.0.20, and unfortunately we got the same error and message back.

Quote by Alex:
Is it possible that there is a syntax problem in the Xml?

or your network connection get lost and MatriX received only partial XML which causes the Xml parse exception?

The presence Xml in the post I posted earlier is the last message sent from the chat server which was shown in both the chat client and through Wireshark. Looking at the xml and comparing it to other messages I can't see anything wrong with it that would cause the exception.

Quote by Alex:
The Xml logs provided get written after MatriX parsed the Xml. We would need the raw data for more diagnosis from a tool like Wireshark or Fiddler.

I've posted the data retrieved from Wireshark in the attachment which shows what was happening when the connection was lost and the error arose which was captured on the server. Message No 4. shows the last message sent from the chat server to the client which was the presence message posted in my previous post. Message No. 5 through No. 11 shows when the connection was lost. Then Message No. 12 is when the client reconnected to the chat server.

Client IP: 10.120.57.109
Chat Server IP: 10.120.52.38

Hopefully this will help us figure out what's happening. Let me know if you have any questions.

Thanks for your help.

The author has attached one file to this post:

WSPackets-9_27_18_5_41.zip 2.5 kBytes
You have no permission to open this file.

Alex

2016-10-01, 03:18 #4

Member since Feb 2003 · 4449 posts · Location: Germany
Group memberships: Administrators, Members

Show profile ·

Link to this post

I see nothing unusual in your log.

Your exception above sais error in position 98305. This must be a huge Xml stanza which I cannot see in your logs.

Alex

Lightbarrier

2016-10-03, 19:37 #5

Member since Jul 2014 · 52 posts
Group memberships: Members

Show profile ·

Link to this post

This is what I was able to retrieve from Wireshark sitting on the chat server and the logs given by the chat client and MatriX through the 'OnReceiveXml' and 'OnSendXml' events.

Is there some way we can retrieve the raw messages through MatriX before they're attempted to be parsed?

Thanks for your help.

Alex

2016-10-04, 10:59 #6

Member since Feb 2003 · 4449 posts · Location: Germany
Group memberships: Administrators, Members

Show profile ·

Link to this post

Quote by Lightbarrier:
Is there some way we can retrieve the raw messages through MatriX before they're attempted to be parsed?

No, but this is where Wireshark, Fiddler and similar tools come in.
The error position in the stacktrace is pretty high, in your log I have not seen a packet of this size.
Does your application send or receive packets of this size?

Alex

Lightbarrier

2016-10-06, 03:08 #7

Member since Jul 2014 · 52 posts
Group memberships: Members

Show profile ·

Link to this post

Based on the test results and what we can see within Wireshark, it seems that the user's roster was expiring within the cache of the chat server. A operation then occurs that causes the roster to again be retrieved from it's data source and rebroadcasted out to the roster's owner. During this process MatriX fails to piece the roster together and ends up raising the error.

These results are shown in the Wireshark pcapng and MatriX trace files in the attachment. Frame 149 through 189 within the pcapng file shows the roster packets that failed to get pieced together, and the MatriX trace files shows the error that was caused by it.

Client IP: 10.120.53.45
Chat Server IP: 10.120.52.38

This error strikes us as odd as the chat client can handle the initial retrieval of the roster upon connecting to the chat server. It's only upon receiving the roster a second time does MatriX fail to receive it and thus causes the error.

We're wondering if MatriX should be able to handle such a situation and if not what are the expected limitations for what can be handled?

Thanks for your help.

The author has attached one file to this post:

ChatTraceFiles_10-05-2016.zip 57 kBytes
You have no permission to open this file.

Alex

2016-10-06, 13:35 #8

Member since Feb 2003 · 4449 posts · Location: Germany
Group memberships: Administrators, Members

Show profile ·

Link to this post

I could not find an invalid packet either in this TCP dump which can cause this exception. All packets I have seen are valid XML.

What I can see is that your complete XMPP session is getting opened again. Its not just retrieving the roster again. It starts from the beginning, so there must be a Close or error of the connection before.

We also wrote a unit test with the 2nd roster you received. MatriX is parsing it without any problems.

Is it possible that the state in your app code is getting confused because of reopening the connection? And calling MatriX functions like Open() when it shouldn't?

Alex

Lightbarrier

2016-10-06, 20:41 #9

Member since Jul 2014 · 52 posts
Group memberships: Members

Show profile ·

Link to this post

Quote by Alex:
I could not find an invalid packet either in this TCP dump which can cause this exception. All packets I have seen are valid XML.

We referenced the wrong frames it's actually frame 11 through frame 107 that shows the roster packets that failed to get pieced together; shortly afterwards around frame (116-121) the chat connection is lost and the error occurs. Within Wireshark using the frame window you can see that frame 11 through 107 are roster packets however, we didn't see a record showing the packets for this roster in a complete xml format within the pcapng file. I've posted some pictures in the attachment highlighting the roster packets that I've talked about to make it more clear.

Quote by Alex:
What I can see is that your complete XMPP session is getting opened again. Its not just retrieving the roster again. It starts from the beginning, so there must be a Close or error of the connection before.

That’s separate from the topic at hand and doesn't involve the problem we're experiencing, but so you understand our logic what is happening is that after MatriX closes the chat session due to the error below the client receives the close event and then automatically reconnects to the chat server. I kept these packets within the pcapng file and these events within the MatriX trace files so you could see that MatriX is correctly receiving the roster on the initial connection to the chat server and that it's only upon receiving the roster a second time does MatriX error out with the error shown below.

Error:
Matrix.OnError(object sender, ExceptionEventArgs error)
error.Exception.GetType(): Matrix.Net.BoshException
error.Exception.Message: BoshException
error.Exception.GetBaseException().GetType(): System.Xml.XmlException
error.Exception.GetBaseException().Message: There is an unclosed literal string. Line 1, position 98305.
Stack Trace of Matrix.OnError: In File

Thanks for your help.

The author has attached one file to this post:

HighlightedWireSharkPacketImages.zip 333.4 kBytes
You have no permission to open this file.

Alex

2016-10-06, 20:56 #10

Member since Feb 2003 · 4449 posts · Location: Germany
Group memberships: Administrators, Members

Show profile ·

Link to this post

Its not MatriX job to put all those TCP frames together. This happens on the OS level. MatriX will retrieve always a complete stanza in a single .NET HTTP WebRequest. In Wireshark you can follow a TcpStream and see the output.
I looked at the packet you mentioned, and there I see an incomplete Http Request with a partial roster response.

So it looks like you lost the connection while the server is sending the response of a WebRequest to MatriX. This explains why you are getting exactly this exception. There is only partial Xml retrieved and the complete response could not bet transmitted. So its expected behavior.

Alex

Lightbarrier

2016-10-06, 21:41 #11

Member since Jul 2014 · 52 posts
Group memberships: Members

Show profile ·

Link to this post

Quote by Alex:
So it looks like you lost the connection while the server is sending the response of a WebRequest to MatriX. This explains why you are getting exactly this exception. There is only partial Xml retrieved and the complete response could not bet transmitted. So its expected behavior.

We understand that only the partial xml was received however, we still have questions about the following.

1. Why does MatriX need to close the connection to the chat server after this? From what we can see after frame 107 the chat server and MatriX continue to exchange presence messages between each other and yet the connection still ends because of the fragmented roster, so why does the connection need to end? Getting disconnected becomes very troublesome for our users when they're in the middle of a conversation and they get logged out for reasons they are unsure of.

2. This may be out of the realm of your expertise, but why does the roster consistently return as a partial response when a chat session is already established, but is then consistently received as a complete response when establishing a new chat session? The number of users in the roster hasn't changed and it's not like the OS or hardware changes either, these reasons are making it hard to understand why the roster comes back fragmented.

Thanks for your help.

Alex

2016-10-06, 22:22 #12

Member since Feb 2003 · 4449 posts · Location: Germany
Group memberships: Administrators, Members

Show profile ·

Link to this post

Quote by Lightbarrier:
1. Why does MatriX need to close the connection to the chat server after this? From what we can see after frame 107 the chat server and MatriX continue to exchange presence messages between each other and yet the connection still ends because of the fragmented roster, so why does the connection need to end? Getting disconnected becomes very troublesome for our users when they're in the middle of a conversation and they get logged out for reasons they are unsure of.

MatriX is getting an Xml stream error. Right now is closes the connection on all Xml stream errors. We would be able to keep the connection alive, but may just loose the packet then which is also bad. I am also not sure what happens on the server side, whether the packet gets resent or not.

Quote by Lightbarrier:
2. This may be out of the realm of your expertise, but why does the roster consistently return as a partial response when a chat session is already established, but is then consistently received as a complete response when establishing a new chat session? The number of users in the roster hasn't changed and it's not like the OS or hardware changes either, these reasons are making it hard to understand why the roster comes back fragmented.

MatriX requests the roster only once. If it gets sent several times then this is something special on your server.
Also there is not need to. When the roster changes during your session then teh server should send you only the changes, and not the complete roster.

Alex

Lightbarrier

2016-10-31, 19:33 #13

Member since Jul 2014 · 52 posts
Group memberships: Members

Show profile ·

Link to this post

Hey Alex,

We managed to update the chat server so it won't send the roster repeatedly and so it won't error out as easily when sending large packets. However, based on one of your responses below we have an update request to try and avoid similar errors that can occur out within the field in the future.

Quote by Alex on 2016-10-06, 22:22:
1475781767

Our idea/feature request to try and prevent similar problems is as follows.

If possible can an property be included into a future update that defaults to false so MatriX performs like it does today, but when set to true MatriX will attempt to keep a connection to the chat server when similar errors occurs that don’t involve the chat server disconnecting from the client after MatriX finishes it’s preparations creating a stable connection.

The above idea is simply one way we can see that would reduce the number of problems that could be experienced within the field. We realize we don’t know how much time an update like this would take, or if you have any better ideas to help resolve these problems. However, we would appreciate it if there was some functionality within MatriX that could allow us to handle problems like the one we experienced more easily in the future so MatriX doesn’t lose its connection to the chat server as easily as we experienced.

Thanks for your assistance.

Alex

2016-11-02, 15:59 #14

Member since Feb 2003 · 4449 posts · Location: Germany
Group memberships: Administrators, Members

Show profile ·

Link to this post

I think you should get a BoshException in the OnError event.
In the BoshException object there is a CanRecover property which you can set to true, MatriX will try to keep the session alive then.

Alex