Non-Blocking MySQL Roster

[Austin Brkich]

I was taking a look at the SQLite Roster Storage that is provided with
DJabberd and was wondering what would be needed to convert it to use
MySQL without blocking. I don't know to much details about the Perl +
MySQL section which is why I am asking for some help. I know you can
setup a MySQL table to use InnoDB which uses row locking instead of
table locking (better for multiple applications accessing the same
table and updating). However from my understanding even if I did use
InnoDB that the Perl DBI stuff will still only execute it one at a
time.

Does anyone have any idea on how I would go about doing this?

[Yann Kerherve]

Post by Austin Brkich
I was taking a look at the SQLite Roster Storage that is provided with
DJabberd and was wondering what would be needed to convert it to use
MySQL without blocking. I don't know to much details about the Perl +
MySQL section which is why I am asking for some help. I know you can
setup a MySQL table to use InnoDB which uses row locking instead of
table locking (better for multiple applications accessing the same
table and updating). However from my understanding even if I did use
InnoDB that the Perl DBI stuff will still only execute it one at a
time.
Does anyone have any idea on how I would go about doing this?

Hi,

There are multiple ways of doing this, you'll probably find some
examples around (LJ auth comes to mind, and there is probably more) of
processing these blocking requests in another process, using gearman.
In this scenario Djabberd instanciates non-blocking Gearman Clients
talking to Gearmand. And this or (those) Gearman server delegates the
work to Gearman workers (yet another processe(s)). When the job is done
geamand asynchonously notifies Djabberd which in the meantime did
other stuff.

Yann

[Austin Brkich]

I was wanting to know if anyone knew the limitations of Perl creating
new threads and such. My idea was to make a sub-routine for Roster &
Auth that can be run in it's own thread and making 2-5 threads (x2, 1
set is for Auth, 2nd set is for Roster) and handing over information
to these new threads. Which would come back with the information and
use the programming already done with DJabberd to reply with the
information.

I know this sounds off but the server I am setting up is going to be
big, real big (1,000+ Users with people logging in an out). The next
thing I am going to be adding is caching of information that is going
to be accessed a lot.

Post by Yann Kerherve

Post by Austin Brkich
I was taking a look at the SQLite Roster Storage that is provided with
DJabberd and was wondering what would be needed to convert it to use
MySQL without blocking. I don't know to much details about the Perl +
MySQL section which is why I am asking for some help. I know you can
setup a MySQL table to use InnoDB which uses row locking instead of
table locking (better for multiple applications accessing the same
table and updating). However from my understanding even if I did use
InnoDB that the Perl DBI stuff will still only execute it one at a
time.
Does anyone have any idea on how I would go about doing this?

Hi,
There are multiple ways of doing this, you'll probably find some
examples around (LJ auth comes to mind, and there is probably more) of
processing these blocking requests in another process, using gearman.
In this scenario Djabberd instanciates non-blocking Gearman Clients
talking to Gearmand. And this or (those) Gearman server delegates the
work to Gearman workers (yet another processe(s)). When the job is done
geamand asynchonously notifies Djabberd which in the meantime did
other stuff.
Yann

[Ask Bjørn Hansen]

Post by Austin Brkich
I was wanting to know if anyone knew the limitations of Perl creating
new threads and such. My idea was to make a sub-routine for Roster &
Auth that can be run in it's own thread and making 2-5 threads (x2, 1
set is for Auth, 2nd set is for Roster) and handing over information
to these new threads.

What Yann described with gearman is the "djabberd way". Perl has threads, but it's much easier to just use multiple processes and event based programming.

Post by Austin Brkich
Which would come back with the information and use the programming already done with DJabberd to reply with the information.
I know this sounds off but the server I am setting up is going to be
big, real big (1,000+ Users with people logging in an out). The next
thing I am going to be adding is caching of information that is going
to be accessed a lot.

A couple thousand users shouldn't be a big deal unless they all login at the very same second.



- ask