Not logged in. · Lost password · Register
Forum: MatriX RSS
FireOnReceive event of XMPPClient class causing the UI thread to block
Avatar
Kiril Tabakov #1
Member since Aug 2015 · 9 posts
Group memberships: Members
Show profile · Link to this post
Subject: XMPP message receiving blocks the UI thread
Hello,
the problem that I'm facing is that when XMPPClient is receiving messages, it's blocking the UI thread. I've run the profiling tools in visual studio and that showed that most of the time that the UI is blocked is taken by FireOnReceive -> Write -> Add.

So far I've tried all variations of setting AutoInvoke (true or false) and setting or not setting the Dispatcher manually (even got as far as adding a new Window of the application and tried using it's dispatcher in my hope to use another thread in order to not block the main UI thread), I've also tried to extend the XMPPClient class and override it's events but didn't got any results with that also.

So a little bit of help from someone from the matrix team would be highly appreciated as the performance issues that UI blocking is creating must be fixed ASAP (some deadlines are coming as usual).

Additional information:
Application is written for WinRT.
Messages that are received could be fairly big (in comparison to chat messages).
I've tried to use also the newer version 2.0.0.4 for WinRT but it was behaving the same way.
Avatar
Alex #2
Member since Feb 2003 · 4322 posts · Location: Germany
Group memberships: Administrators, Members
Show profile · Link to this post
when AutoInvoke is true then MatriX automatically tries to find your dispatcher and is invoking all events automatically for you.
This is what most users want and save you lots of invoke code.

When AutoInvoke is false then MatriX does not invoke the events at all, and its up to you to invoke them.

We never heard of any performance problems from our users, and when AutoInvoke is false it should not block your applications.

Also for very big messages its recommended to split them in multiple chunks, because they will block your XMPP stream.
Avatar
Kiril Tabakov #3
Member since Aug 2015 · 9 posts
Group memberships: Members
Show profile · Link to this post
Hi, Alex.

Thank you for the response, we've been doing some more testing since that to try to isolate the problem a narrow down the cause.

So first of all I have to apologize because I've used somewhat incorrect term, the UI is not completely blocked but it's performance is greatly hindered, in the moment of receiving xmpp message the processor usage is rising a lot and somehow (even with AutoInvoke = false) this code seems to be running on some of the UI threads or at least it is slowing them somehow.

We disabled the additional processing of the messages, deserialization, decoding, so we are only receiving them and it still hinders the performance greatly when we try to do some UI work (navigating from one page to another).

What I've noticed is that we have messages sometimes with the length of more than half a million symbols (700 000) (maybe even bigger I'm continuing the tests), and for this exact message for example has around 9000 lines. So I was wondering if this is supported size of message, and also if it's normal to have the message of so many lines with 76 symbols of length.

Here are the results of the performance tools

http://s23.postimg.org/dnj39ekez/Performance_Detailed_Repo…
http://s15.postimg.org/ot5hziue3/Performance_Full_Report.png
http://s4.postimg.org/sdud0qb9p/Performance_Detailed_Repor…

Summary:

1: Do you think matrix fire on receive is happening somehow on the UI thread (some of them)?

2: Do you think the message length is ok for xmpp to parse?

3: Do you think the newlines could be causing performance issue for the xml parser of matrix or it's normal?

3.1: If not, is it possible to change the format somehow?

4: Do you have any idea if and how Matrix could be impacting the UI performance in a different manner?

It would be very nice if you can respond timely, and we'll appreciate any advice and information
, Thanks.
Avatar
Kiril Tabakov #4
Member since Aug 2015 · 9 posts
Group memberships: Members
Show profile · Link to this post
Some information of the report timeline:

You could see what happens on first install of the application, in the beginning the UI is responsive, than it starts receiving data from the server and you could see the increase of the load on the UI thread is caused only by application code. Beneath you could see that the Platform specific project is not taking up the time but the attached portable class library, so in the detailed report you could see which method emerges as the bottleneck at the particular time slice of the report.

You could see at the end when all data is received and the server stops sending how the application is behaving normally.

All the methods handling the processing are asynchronous and even if we turn off the processing the delay is there when XmppClient is receiving messages.
Avatar
Alex #5
Member since Feb 2003 · 4322 posts · Location: Germany
Group memberships: Administrators, Members
Show profile · Link to this post
1) When AutoInvoke is false MatriX should not hop on your Ui thread.

2) No its not. XMPP is not designed for such huge messages. Such huge messages are received in multiple packets over the tcp layer, and have to be assembled in the XML parser. The parser must also find the message end which is not trivial depending on how your message Xml in structured. You should send huge messages in smaller chunks.
Avatar
Kiril Tabakov #6
Member since Aug 2015 · 9 posts
Group memberships: Members
Show profile · Link to this post
Hi, Alex.

2: Could you elaborate on what is the preferred message size (eg. 100kb, 10kb or xxxx number of symbols etc), that are not going to work OK with Matrix SDK and XMPP in general so that we know what type of size we're targeting?

4: Any idea if we are to expect some underwater rocks or size of messages should suffice?

Thanks
Avatar
Alex #7
Member since Feb 2003 · 4322 posts · Location: Germany
Group memberships: Administrators, Members
Show profile · Link to this post
Quote by Kiril Tabakov:
2: Could you elaborate on what is the preferred message size (eg. 100kb, 10kb or xxxx number of symbols etc), that are not going to work OK with Matrix SDK and XMPP in general so that we know what type of size we're targeting?

there is no preferred or suggested maximum size. This hardly depends on your use case, payloads, requirements, network, traffic and software design. Usually max ~10K is what many people suggest. But I have seen use cases where message sizes of multiple MBs are no problem at all, and other use cases where 10K is too big.

The Xml parser in MatriX is extremely lightweight and fast in general. So I have no idea right now where the problems are coming from you are facing.

XMPP is using 2 Xml stream one 1 async TCP connection. One stream for all incoming packets and another one for all outgoing packets. So its like a FIFO buffer. While its parses and a message it cannot receive another one in parallel, which may be a problem when you have huge messages which can block the stream for a while.

Events like OnMessage get initiated by the stream parser in MatriX. When you want high throughput then you should consider to put your message just in a queue in the OnMessage handler and then process them from other threads.


Quote by Kiril Tabakov:
4: Any idea if we are to expect some underwater rocks or size of messages should suffice?

Sorry, I don't understand this question, can you please elaborate?

Alex
Avatar
Kiril Tabakov #8
Member since Aug 2015 · 9 posts
Group memberships: Members
Show profile · Link to this post
Hi Alex,

Thanks again for the information, and for the suggestion to put processing in queue, but our current problem is happening before OnMessage it's somewhere in the steps before that (the parsing I assume), and our processing is happening on another thread already (also measured it and it's only taking few milliseconds as it's async, got as far as disabling it all together and still had the problem).

As far as

Quote by Kiril Tabakov:
4: Any idea if we are to expect some underwater rocks or size of messages should suffice?

Sorry, I don't understand this question, can you please elaborate?

Alex

I was asking if it seems likely that we encounter some other problem (other than the message size) or it is rather unlikely. I know it is too much of a general question and not easy to answer without a context but if anything comes to mind please share. (I'm currently working on reducing messages length and most probably will separate them. After that I'll share what happened).
Avatar
Kiril Tabakov #9
Member since Aug 2015 · 9 posts
Group memberships: Members
Show profile · Link to this post
Subject: Result and thoughts
Hi Alex,

As promised I'm writing to inform you and all users that may eventually run into problem like ours, what the results are from splitting up longer messages to smaller chunks. I don't have a lot of time to experiment with varying options and also do not have that much freedom on the server side (that's why it took that much time to get back with the results).

So a short and simplified description on what we did to fix the UI blocking:

On the server side

1. Added compression to reduce the message size 3-5 times (not much testing but might vary depending on actual messages)

2. Divide messages larger than defined size (we worked with suggested size by you for now 1024 bytes * 10 for message body with a little bit more info in total ~ 10kb) to chunks ~ 10kb and send them as separate messages to the client.

On the client side

1. Added some classes to handle the reassembling of message chunks to one complete message. Nothing fancy just grouping messages by randomly generated id, and arranging them by number in a sorted dictionary. Then sending them to process queue (as suggested by Alex) but at the moment that part is not used as there is no problem with processing speed and it is (really) running on the background so it doesn't block anything.

2. Shorter messages are processed the same way as before, while larger go through the reassembling process and then are processed the exact same way as before.

The RESULT

Much much higher level of responsiveness of the UI (what was our hope and what we think is happening is that the UI now has more time to respond as the messages are small and are parsed quicker than before so the UI is responsive between messages).

However at some moments the UI is still blocked (for very short amounts of time) and I imagine for the same reason as before (message parsing is still the most likely candidate in my oppinion). Maybe if we further reduce message size and get rid of something unneeded we might get better performance but I think the cause is still there.

Significant difference in cpu usage - from around 40-45% (measured on laptop with Intel Core I7-4500U processor) to ~10% in the peak moments of parsing data.

TO CONSIDER:

So again i think it all might have something to do with the parsing and string appending as I can't think of different reason for excessive CPU usage at that exact moment (of message receiving). However I'm not the author of the parser's code nor any other part of the Matrix SDK so I might be terribly wrong, but I really think it is the correct thing to do, for Alex or someone else on the team - to take a look at the code and to check out for possible causes of blocking or ineffective parsing at our initial use case - very long message received at once (some of the messages were with more than 700 000 symbols and have been received with newlines at ~70th symbol so around 10 000 newlines).

After all if I'm wrong, everything's OK and no harm is done, but if there is a bug in the parser that could easily be fixed and save some people the trouble to work around it in the future, then it might as well be worth a few hours of investigation on part of the Matrix SDK team.

Let's hope the results are enough to satisfy the management and happy coding to you all ;)
Avatar
Alex #10
Member since Feb 2003 · 4322 posts · Location: Germany
Group memberships: Administrators, Members
Show profile · Link to this post
Hello Kiril,

great to hear that you were able to improve the speed significant with our suggestions.
I still don't understand why MatriX blocks your UI thread. All the MatriX events get invoked internal by read operations of the tcp/ip socket. And all socket operations are asynchronous and non blocking. They should be on different threads managed by the .NET Framework.

Have you tried a POC on the Full .NET stack and not WinRT? Because WinRT has also some restrictions.

I have some ideas of tweaks which may help you for further improvements, like increasing internal buffer size. When you contact us directly by Email we could compile you a test version and see if it helps.

Also keep in mind that there is no software design which fits best for all use cases. If you wanna get the maximum out of MatriX you should consider going with the source version which would allow you further internal tweaks and profiling when required.

Alex
Avatar
Kiril Tabakov #11
Member since Aug 2015 · 9 posts
Group memberships: Members
Show profile · Link to this post
Hi Alex,

I've not tried any POC projects (even on WinRT) as I didn't have much time, also haven't tried .Net version (the server is written in Java).

After I finish my current tasks if there is some time I might do that, but if the results are satisfactory in the current state I doubt we'll invest the time right now. At some point we might get to full C# implementation as Xamarin options are on the horizon but nothing is set in stone still.

I agree completely on the fact that there is no "one size fits all" solution in software development and I'm not asking/demanding such thing.

When I finish my current tasks I'll contact you by Email so that we could check how the buffer size (or the other ideas) could influence the performance in our specific case.

Thanks again for the suggestions.
Close Smaller – Larger + Reply to this post:
Verification code: VeriCode Please enter the word from the image into the text field below. (Type the letters only, lower case is okay.)
Smileys: :-) ;-) :-D :-p :blush: :cool: :rolleyes: :huh: :-/ <_< :-( :'( :#: :scared: 8-( :nuts: :-O
Special characters:
Forum: MatriX RSS