Not logged in. · Lost password · Register
Forum: MatriX and XmppDotNet RSS
MatriX.vNext XmppClient thread leakage when creating new XmppClient objects
Avatar
sldr 2018-02-28, 06:18 #1
Member since Feb 2018 · 15 posts
Group memberships: Members
Show profile · Link to this post
Subject: MatriX.vNext XmppClient thread leakage when creating new XmppClient objects
I keep getting thread leakage from DotNetty.Common.dll!DotNetty.Common.Concurrency.SingleThreadEventExecutor.PollTask when I recreate XmppClient even when calling DisconnectAsync() (or DisconnectAsync(false)) before recreation. This is happening with 1.0.6 and the CI builds listed below:

Id                                  Versions                                 Description
--                                  --------                                 -----------
Matrix.vNext                        {2.0.0-ci-18058-5}                       MatriX vNext
Matrix.vNext.Extensions             {2.0.0-ci-18058-5}                       MatriX vNext

Here is the list of threads:
Not Flagged    >    0x00003700    0x00000001    Main Thread    Main Thread    EMailIM.exe!EMailIM.Program.Main    Normal
Not Flagged        0x00003B7C    0x00000003    Worker Thread    <No Name>    DotNetty.Common.dll!DotNetty.Common.Concurrency.SingleThreadEventExecutor.PollTask    Normal
Not Flagged        0x00002408    0x00000004    Worker Thread    <No Name>    DotNetty.Common.dll!DotNetty.Common.Concurrency.SingleThreadEventExecutor.PollTask    Normal
Not Flagged        0x000093AC    0x00000005    Worker Thread    <No Name>    DotNetty.Common.dll!DotNetty.Common.Concurrency.SingleThreadEventExecutor.PollTask    Normal
Not Flagged        0x000084BC    0x00000006    Worker Thread    <No Name>    DotNetty.Common.dll!DotNetty.Common.Concurrency.SingleThreadEventExecutor.PollTask    Normal
Not Flagged        0x00005474    0x00000007    Worker Thread    <No Name>    DotNetty.Common.dll!DotNetty.Common.Concurrency.SingleThreadEventExecutor.PollTask    Normal
Not Flagged        0x000082E0    0x00000008    Worker Thread    <No Name>    DotNetty.Common.dll!DotNetty.Common.Concurrency.SingleThreadEventExecutor.PollTask    Normal
Not Flagged        0x000097A8    0x00000009    Worker Thread    <No Name>    DotNetty.Common.dll!DotNetty.Common.Concurrency.SingleThreadEventExecutor.PollTask    Normal
Not Flagged        0x00001A38    0x0000000A    Worker Thread    <No Name>    DotNetty.Common.dll!DotNetty.Common.Concurrency.SingleThreadEventExecutor.PollTask    Normal
Not Flagged        0x00005BDC    0x0000000B    Worker Thread    <No Name>    DotNetty.Common.dll!DotNetty.Common.Concurrency.SingleThreadEventExecutor.PollTask    Normal
Not Flagged        0x000055E8    0x0000000C    Worker Thread    <No Name>    DotNetty.Common.dll!DotNetty.Common.Concurrency.SingleThreadEventExecutor.PollTask    Normal
Not Flagged        0x00004958    0x0000000D    Worker Thread    <No Name>    DotNetty.Common.dll!DotNetty.Common.Concurrency.SingleThreadEventExecutor.PollTask    Normal
Not Flagged        0x00001BD4    0x0000000E    Worker Thread    <No Name>    DotNetty.Common.dll!DotNetty.Common.Concurrency.SingleThreadEventExecutor.PollTask    Normal
Not Flagged        0x00003568    0x0000000F    Worker Thread    <No Name>    DotNetty.Common.dll!DotNetty.Common.Concurrency.SingleThreadEventExecutor.PollTask    Normal
Not Flagged        0x000040A4    0x00000010    Worker Thread    <No Name>    DotNetty.Common.dll!DotNetty.Common.Concurrency.SingleThreadEventExecutor.PollTask    Normal
Not Flagged        0x0000986C    0x00000011    Worker Thread    <No Name>    DotNetty.Common.dll!DotNetty.Common.Concurrency.SingleThreadEventExecutor.PollTask    Normal
Not Flagged        0x00008E78    0x00000012    Worker Thread    <No Name>    DotNetty.Common.dll!DotNetty.Common.Concurrency.SingleThreadEventExecutor.PollTask    Normal
Not Flagged        0x00008054    0x00000017    Worker Thread    Worker Thread        Normal
Not Flagged        0x00007070    0x00000018    Worker Thread    <No Name>    DotNetty.Common.dll!DotNetty.Common.Concurrency.SingleThreadEventExecutor.PollTask    Normal
Not Flagged        0x00007690    0x0000001A    Worker Thread    <No Name>    DotNetty.Common.dll!DotNetty.Common.Concurrency.SingleThreadEventExecutor.PollTask    Normal
Not Flagged        0x000053F4    0x0000001B    Worker Thread    <No Name>    DotNetty.Common.dll!DotNetty.Common.Concurrency.SingleThreadEventExecutor.PollTask    Normal
Not Flagged        0x00006CA0    0x0000001C    Worker Thread    <No Name>    DotNetty.Common.dll!DotNetty.Common.Concurrency.SingleThreadEventExecutor.PollTask    Normal
Not Flagged        0x0000207C    0x0000001D    Worker Thread    <No Name>    DotNetty.Common.dll!DotNetty.Common.Concurrency.SingleThreadEventExecutor.PollTask    Normal
Not Flagged        0x00007994    0x0000001E    Worker Thread    <No Name>    DotNetty.Common.dll!DotNetty.Common.Concurrency.SingleThreadEventExecutor.PollTask    Normal
Not Flagged        0x00009924    0x0000001F    Worker Thread    <No Name>    DotNetty.Common.dll!DotNetty.Common.Concurrency.SingleThreadEventExecutor.PollTask    Normal
Not Flagged        0x000061E0    0x00000020    Worker Thread    <No Name>    DotNetty.Common.dll!DotNetty.Common.Concurrency.SingleThreadEventExecutor.PollTask    Normal
Not Flagged        0x00002004    0x00000021    Worker Thread    <No Name>    DotNetty.Common.dll!DotNetty.Common.Concurrency.SingleThreadEventExecutor.PollTask    Normal
Not Flagged        0x000071BC    0x00000022    Worker Thread    <No Name>    DotNetty.Common.dll!DotNetty.Common.Concurrency.SingleThreadEventExecutor.PollTask    Normal
Not Flagged        0x00008B20    0x00000023    Worker Thread    <No Name>    DotNetty.Common.dll!DotNetty.Common.Concurrency.SingleThreadEventExecutor.PollTask    Normal
Not Flagged        0x000020A0    0x00000024    Worker Thread    <No Name>    DotNetty.Common.dll!DotNetty.Common.Concurrency.SingleThreadEventExecutor.PollTask    Normal
Not Flagged        0x00007684    0x00000025    Worker Thread    <No Name>    DotNetty.Common.dll!DotNetty.Common.Concurrency.SingleThreadEventExecutor.PollTask    Normal
Not Flagged        0x0000A168    0x00000026    Worker Thread    <No Name>    DotNetty.Common.dll!DotNetty.Common.Concurrency.SingleThreadEventExecutor.PollTask    Normal
Not Flagged        0x000037B4    0x00000027    Worker Thread    <No Name>    DotNetty.Common.dll!DotNetty.Common.Concurrency.SingleThreadEventExecutor.PollTask    Normal
Not Flagged        0x000071F4    0x00    Worker Thread    GC Finalizer Thread        Normal
Not Flagged        0x0000A204    0x00000028    Worker Thread    <No Name>    DotNetty.Common.dll!DotNetty.Common.Concurrency.SingleThreadEventExecutor.PollTask    Normal

Call stacks in part 2 of this message because 10000 char post limit.

It seems to create 16 threads for each new XmppClient I create. My system has 8 logical processors btw and that probably has something to do with 16 threads per XmppClient.

If you need more info let me know.

PS: Thanks for adding the TimeIq class.

Laters,
SLDR
Avatar
sldr 2018-02-28, 06:20 #2
Member since Feb 2018 · 15 posts
Group memberships: Members
Show profile · Link to this post
Subject: Part 2
Here is a Call Stack from a random thread (all that I checked (about half) were the same):

>    mscorlib.dll!System.Threading.ManualResetEventSlim.Wait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken) Line 669    C#    Symbols loaded.
     DotNetty.Common.dll!DotNetty.Common.Concurrency.SingleThreadEventExecutor.PollTask()    Unknown    No symbols loaded.
     DotNetty.Common.dll!DotNetty.Common.Concurrency.SingleThreadEventExecutor.RunAllTasks(DotNetty.Common.PreciseTimeSpan timeout)    Unknown    No symbols loaded.
     DotNetty.Common.dll!DotNetty.Common.Concurrency.SingleThreadEventExecutor.Loop.AnonymousMethod__25_0()    Unknown    No symbols loaded.
     mscorlib.dll!System.Threading.Tasks.Task.Execute() Line 2498    C#    Symbols loaded.
     mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Line 954    C#    Symbols loaded.
     mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Line 902    C#    Symbols loaded.
     mscorlib.dll!System.Threading.Tasks.Task.ExecuteWithThreadLocal(ref System.Threading.Tasks.Task currentTaskSlot) Line 2827    C#    Symbols loaded.
     mscorlib.dll!System.Threading.Tasks.Task.ExecuteEntry(bool bPreventDoubleExecution) Line 2767    C#    Symbols loaded.
     mscorlib.dll!System.Threading.Tasks.Task.ScheduleAndStart(bool needsProtection) Line 1946    C#    Symbols loaded.
     mscorlib.dll!System.Threading.Tasks.Task.InternalStartNew(System.Threading.Tasks.Task creatingTask, System.Delegate action, object state, System.Threading.CancellationToken cancellationToken, System.Threading.Tasks.TaskScheduler scheduler, System.Threading.Tasks.TaskCreationOptions options, System.Threading.Tasks.InternalTaskOptions internalOptions, ref System.Threading.StackCrawlMark stackMark) Line 1294    C#    Symbols loaded.
     mscorlib.dll!System.Threading.Tasks.TaskFactory.StartNew(System.Action action, System.Threading.CancellationToken cancellationToken, System.Threading.Tasks.TaskCreationOptions creationOptions, System.Threading.Tasks.TaskScheduler scheduler) Line 401    C#    Symbols loaded.
     DotNetty.Common.dll!DotNetty.Common.Concurrency.SingleThreadEventExecutor.Loop()    Unknown    No symbols loaded.
     DotNetty.Common.dll!DotNetty.Common.Concurrency.XThread.CreateLongRunningTask.AnonymousMethod__0()    Unknown    No symbols loaded.
     mscorlib.dll!System.Threading.Tasks.Task.Execute() Line 2498    C#    Symbols loaded.
     mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Line 954    C#    Symbols loaded.
     mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Line 902    C#    Symbols loaded.
     mscorlib.dll!System.Threading.Tasks.Task.ExecuteWithThreadLocal(ref System.Threading.Tasks.Task currentTaskSlot) Line 2827    C#    Symbols loaded.
     mscorlib.dll!System.Threading.Tasks.Task.ExecuteEntry(bool bPreventDoubleExecution) Line 2767    C#    Symbols loaded.
     mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Line 954    C#    Symbols loaded.
     mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Line 902    C#    Symbols loaded.
     mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) Line 891    C#    Symbols loaded.
     mscorlib.dll!System.Threading.ThreadHelper.ThreadStart(object obj) Line 93    C#    Symbols loaded.

Thanks,
SLDR
(Stephen L. De Rudder)
Avatar
Alex 2018-02-28, 13:58 #3
Member since Feb 2003 · 4449 posts · Location: Germany
Group memberships: Administrators, Members
Show profile · Link to this post
Hello Stephen,

thanks for reporting.

Can you please try the latest 2.0 build from the myget feed?
XmppConnection implements IDisposable now which will shutdown the DotNetty event loop group.

Alex
Avatar
sldr 2018-02-28, 14:07 #4
Member since Feb 2018 · 15 posts
Group memberships: Members
Show profile · Link to this post
I realize I forgot to give you the key information that causes the thread leakage. I go to my IM server and kill my session. That causes my code to create a new XmppClient object and the 16 threads from the previous XmppClient stay around forever (even when DisconnectAsync() is called). I have included the full program to reproduce:

  1. using Matrix;
  2. using Matrix.Extensions.Client.Presence;
  3. using Matrix.Network;
  4. using Matrix.Xml;
  5. using Matrix.Xmpp;
  6. using Matrix.Xmpp.Chatstates;
  7. using Matrix.Xmpp.Client;
  8. using System;
  9. using System.Linq;
  10. using System.Net.Mail;
  11. using System.Net.Security;
  12. using System.Reactive.Linq;
  13. using System.Security.Cryptography.X509Certificates;
  14. using System.Threading;
  15.  
  16. namespace EMailIM
  17. {
  18.  class Program : ICertificateValidator
  19.  {
  20.  
  21.     public class HUP
  22.     {
  23.       public string hostname;
  24.       public string username;
  25.       public string password;
  26.       public HUP(string h, string u, string p)
  27.       {
  28.         hostname = h;
  29.         username = u;
  30.         password = p;
  31.       }
  32.     }
  33.  
  34.     private XmppClient x = null;
  35.     private string hostname;
  36.     private string username;
  37.     private string password;
  38.  
  39.     public Program(string h, string u, string p)
  40.     {
  41.       this.hostname = h;
  42.       this.username = u;
  43.       this.password = p;
  44.     }
  45.    
  46.     public void DoWork()
  47.     {
  48.       x = new XmppClient();
  49.       x.XmppDomain = hostname;
  50.       x.Username = username;
  51.       x.Password = password;
  52.       x.Resource = "EMailIM";
  53.       x.CertificateValidator = this;
  54.       SetupSubscribes();
  55.       x.ConnectAsync().Wait();
  56.       x.XmppSessionStateObserver.Subscribe(ss =>
  57.       {
  58.         DoWork2(ss);
  59.       });
  60.       x.SendPresenceAsync(Show.Chat, "EMailIM", 22).Wait();
  61.     }
  62.  
  63.     private void DoWork2(SessionState ss)
  64.     {
  65.       Console.WriteLine("SessionState: " + ss.ToString());
  66.       if (ss == SessionState.Disconnected) {
  67.         Thread t = new Thread(DisconnectReconnect);
  68.         t.Start();
  69.       }
  70.     }
  71.  
  72.     private void DisconnectReconnect()
  73.     {
  74.       try {
  75.         // x.DisconnectAsync().Wait(); // doesn't cleanup threads
  76.         x.DisconnectAsync(false).Wait(); // doesn't cleanup threads
  77.       }
  78.       catch (Exception e) {
  79.         // Just ignore all exceptions when trying to disconnect
  80.         if (e.InnerException != null) {
  81.           Console.WriteLine("Exception: " + e.ToString() + "\n  " + e.InnerException.ToString());
  82.         } else {
  83.           Console.WriteLine("Exception: " + e.ToString());
  84.         }
  85.       }
  86.       Program.Reconnect(new HUP(hostname, username, password));
  87.     }
  88.  
  89.     private void SetupSubscribes()
  90.     {
  91.       x.XmppXElementStreamObserver.Where(el => el.OfType<Message>() && el.Cast<Message>().Chatstate == Chatstate.Active).Subscribe(el =>
  92.       {
  93.         Message m = el.Cast<Message>();
  94.         Console.WriteLine(el.ToString());
  95.         SendEMail(m.From, m.Body);
  96.       });
  97.       x.XmppXElementStreamObserver.Where(el => el.OfType<Iq>() && el.Cast<Iq>().Type == IqType.Get && el.Cast<Iq>().Query.OfType<Matrix.Xmpp.Version.Version>()).Subscribe(el =>
  98.       {
  99.         Iq iq = el.Cast<Iq>();
  100.         VersionIq resIq = Factory.GetElement<VersionIq>();
  101.         resIq.Id = iq.Id;
  102.         resIq.From = iq.To;
  103.         resIq.To = iq.From;
  104.         resIq.Type = IqType.Result;
  105.         Matrix.Xmpp.Version.Version v = new Matrix.Xmpp.Version.Version();
  106.         v.Name = "SLDR EMailIM";
  107.         v.Os = "Windows";
  108.         v.Ver = "SLDR EMailIM V0.22";
  109.         resIq.Version = v;
  110.         x.SendAsync(resIq).Wait();
  111.       });
  112.       x.XmppXElementStreamObserver.Where(el => el.OfType<Iq>() && el.Cast<Iq>().Type == IqType.Get && el.Cast<Iq>().Query.OfType<Matrix.Xmpp.Time.Time>()).Subscribe(el =>
  113.       {
  114.         Iq iq = el.Cast<Iq>();
  115.         IqQuery<Matrix.Xmpp.Time.Time> resIq = new IqQuery<Matrix.Xmpp.Time.Time>();
  116.         resIq.Id = iq.Id;
  117.         resIq.From = iq.To;
  118.         resIq.To = iq.From;
  119.         resIq.Type = IqType.Result;
  120.         resIq.Query.DateTime = DateTime.Now;
  121.         x.SendAsync(resIq).Wait();
  122.       });
  123. #if DEBUG
  124.       x.XmppXElementStreamObserver.Subscribe(el => {
  125.         Console.WriteLine(el.ToString());
  126.       });
  127. #endif
  128.     }
  129.  
  130.     public void SendEMail(string imfrom, string message)
  131.     {
  132.       // send an email
  133.     }
  134.  
  135.     private static Program p;
  136.  
  137.     static void Main(string[] args)
  138.     {
  139.       Console.WriteLine("Please enter your XmppDomain");
  140.       string hostname = Console.ReadLine();
  141.       Console.WriteLine("Please enter your username");
  142.       string username = Console.ReadLine();
  143.       Console.WriteLine("Please enter your password");
  144.       string password = ReadPassword();
  145.       p = new Program(hostname, username, password);
  146.       p.DoWork();
  147.       while (true) {
  148.         Thread.Sleep(1000 * 60 * 10);
  149.       }
  150.     }
  151.  
  152.     public static void Reconnect(object o)
  153.     {
  154.       Console.WriteLine("Reconnecting delay start!");
  155.       Thread.Sleep(1000 * 10);
  156.       Console.WriteLine("Reconnecting!");
  157.       HUP up = (HUP)o;
  158.       p = new Program(up.hostname, up.username, up.password);
  159.       p.DoWork();
  160.       Console.WriteLine("Reconnected!");
  161.     }
  162.  
  163.     public static string ReadPassword()
  164.     {
  165.       string password = "";
  166.       ConsoleKeyInfo info = Console.ReadKey(true);
  167.       while (info.Key != ConsoleKey.Enter) {
  168.         if (info.Key != ConsoleKey.Backspace) {
  169.           Console.Write("*");
  170.           password += info.KeyChar;
  171.         } else if (info.Key == ConsoleKey.Backspace) {
  172.           if (!string.IsNullOrEmpty(password)) {
  173.             // remove one character from the list of password characters
  174.             password = password.Substring(0, password.Length - 1);
  175.             // get the location of the cursor
  176.             int pos = Console.CursorLeft;
  177.             // move the cursor to the left by one character
  178.             Console.SetCursorPosition(pos - 1, Console.CursorTop);
  179.             // replace it with space
  180.             Console.Write(" ");
  181.             // move the cursor to the left by one character again
  182.             Console.SetCursorPosition(pos - 1, Console.CursorTop);
  183.           }
  184.         }
  185.         info = Console.ReadKey(true);
  186.       }
  187.       // add a new line because user pressed enter at the end of their password
  188.       Console.WriteLine();
  189.       return password;
  190.     }
  191.  
  192.  public bool RemoteCertificateValidationCallback(object sender, X509Certificate certificate, X509Chain chain, SslPolicyErrors sslPolicyErrors)
  193.     {
  194.       return true;
  195.     }
  196.  }
  197. }

Thanks,
SLDR
This post was edited on 2018-02-28, 14:08 by Alex.
Avatar
Alex 2018-02-28, 14:10 #5
Member since Feb 2003 · 4449 posts · Location: Germany
Group memberships: Administrators, Members
Show profile · Link to this post
see my post above, there is a new build which should fix this for you.

But you also don't have to recreate the XmppClient. You could reuse the existing client and just call Connect again to reconnect.
Either way should work fine with the latest builds.

Alex
Avatar
sldr 2018-02-28, 14:13 #6
Member since Feb 2018 · 15 posts
Group memberships: Members
Show profile · Link to this post
In reply to post #3
Did you change your reply? I could swear you had asked for more info. Its early so maybe I read something else. Sorry for confusion.

I will try the new code and let you know.

Thanks,
SLDR
Avatar
sldr 2018-02-28, 14:51 #7
Member since Feb 2018 · 15 posts
Group memberships: Members
Show profile · Link to this post
It works!!!  :-)

The change to call dispose:
  1.     private void DisconnectReconnect()
  2.     {
  3.       try {
  4.         x.DisconnectAsync().Wait();
  5.       }
  6.       catch (Exception e) {
  7.         // Just ignore all exceptions when trying to disconnect
  8.         Console.WriteLine("Disconnect Exception: " + e.ToString());
  9.       }
  10.       try {
  11.         x.Dispose();
  12.       }
  13.       catch (Exception e) {
  14.         // Just ignore all exceptions when trying to dispose
  15.         Console.WriteLine("Dispose Exception: " + e.ToString());
  16.       }
  17.       Program.Reconnect(new HUP(hostname, username, password));
  18.     }

Thanks!

BTW: If I try to reuse the XmppClient instance I can not get the XmppXElementStreamObserver to fire subscribers. Even if I rerun SetupSubscribes (from my code example). The XmppSessionStateObserver will keep its subscribers and continue to fire. I have not tested with your new IDisposable code yet, but I can not call Dispose() and try to continue to use the object instance; so I am not sure that will fix trying to use XmppClient instance to reconnect. Let me know if you would like me to modify the example to reproduce the XmppClient reconnect issue.

Thanks once again for all your help and QUICK response.

Laters,
SLDR
(Stephen L. De Rudder)

PS: You don't need to edit my post code=csharp now ;-) (ok maybe for other reasons...)
Avatar
Alex 2018-02-28, 15:33 #8
Member since Feb 2003 · 4449 posts · Location: Germany
Group memberships: Administrators, Members
Show profile · Link to this post
Quote by sldr:
BTW: If I try to reuse the XmppClient instance I can not get the XmppXElementStreamObserver to fire subscribers. Even if I rerun SetupSubscribes (from my code example). The XmppSessionStateObserver will keep its subscribers and continue to fire. I have not tested with your new IDisposable code yet, but I can not call Dispose() and try to continue to use the object instance; so I am not sure that will fix trying to use XmppClient instance to reconnect. Let me know if you would like me to modify the example to reproduce the XmppClient reconnect issue.

yes, we are working on those. This is one of the last outstanding items on the v2 roadmap. Its calling OnComplete on the subscription which it shoulnd't. You can expect a fix shortly.
In the meantime you can just dispose and create a new XmppClient

Quote by sldr:
PS: You don't need to edit my post code=csharp now ;-) (ok maybe for other reasons...)

 :-D
Avatar
Alex 2018-02-28, 21:16 #9
Member since Feb 2003 · 4449 posts · Location: Germany
Group memberships: Administrators, Members
Show profile · Link to this post
there is a new build on myget. XmppXElementStreamObserver observer works as expected now also after reconnect.

Alex
Avatar
sldr 2018-03-01, 02:47 #10
Member since Feb 2018 · 15 posts
Group memberships: Members
Show profile · Link to this post
It worked great!!!

THANK you.

Laters,
SLDR
Avatar
Alex 2018-03-01, 11:07 #11
Member since Feb 2003 · 4449 posts · Location: Germany
Group memberships: Administrators, Members
Show profile · Link to this post
Thanks for your feedback.

Alex
Close Smaller – Larger + Reply to this post:
Verification code: VeriCode Please enter the word from the image into the text field below. (Type the letters only, lower case is okay.)
Smileys: :-) ;-) :-D :-p :blush: :cool: :rolleyes: :huh: :-/ <_< :-( :'( :#: :scared: 8-( :nuts: :-O
Special characters:

Forum: MatriX and XmppDotNet RSS