[Firebird-devel] TcpRemoteBufferSize

[Dimitry Sibiryakov]

Hello, All.

IIRC, couple of years ago Alex tested effect of increasing TcpRemoteBufferSize to
network performance and the tests showed that size 1M results in significant performance
boost for applications that fetch large volumes of data.
But in current firebird.conf in 3.0 I still see ancient default 8192 with upper limit
32k. Why?

[Alex Peshkoff]

Post by Dimitry Sibiryakov
Hello, All.
IIRC, couple of years ago Alex tested effect of increasing TcpRemoteBufferSize

Telling true I do not remember it...
But quite possible.

Post by Dimitry Sibiryakov
to
network performance and the tests showed that size 1M results in significant performance
boost for applications that fetch large volumes of data.
But in current firebird.conf in 3.0 I still see ancient default 8192 with upper limit
32k. Why?

[Dimitry Sibiryakov]

Post by Alex Peshkoff
Telling true I do not remember it...
But quite possible.

Most likely my memory doesn't serve me well. Google shows me only thread where Carlos
H. Cantu tested it up to current limit and got 3x speed up, then DE suggested to test it
beyond the limit and then all was over.

[Leyne, Sean]

Post by Dimitry Sibiryakov

Post by Alex Peshkoff
Telling true I do not remember it...
But quite possible.

Most likely my memory doesn't serve me well. Google shows me only
thread where Carlos H. Cantu tested it up to current limit and got 3x speed
up, then DE suggested to test it beyond the limit and then all was over.

Is this buffer size setting configurable thru a connection/API setting?


Sean

[Dimitry Sibiryakov]

Post by Leyne, Sean
Is this buffer size setting configurable thru a connection/API setting?

AFAIK, no.

[Alex Peshkoff]

Post by Dimitry Sibiryakov

Post by Leyne, Sean
Is this buffer size setting configurable thru a connection/API setting?

AFAIK, no.

It's possible to make it configurable in FB3 at the client side. At the
server side we must first allocate some buffer to be able to know
something from that connection.
If we find out that this setting does give some serious effect - even at
server side we can realloc that buffer, but before doing this let's find
a case when increasing buffer size is really useful. And let's also not
forget that making buffer too big at server with that size controllable
by connection is not good from security POV.

[marius adrian popa]

Isn't the recent kernels do autotuning for tcp buffer sizes ?
http://fasterdata.es.net/host-tuning/background/#t1

The max size still can be usefull but we need to see it with real tests

Post by Alex Peshkoff

Post by Dimitry Sibiryakov

Post by Leyne, Sean
Is this buffer size setting configurable thru a connection/API setting?

AFAIK, no.

It's possible to make it configurable in FB3 at the client side. At the
server side we must first allocate some buffer to be able to know
something from that connection.
If we find out that this setting does give some serious effect - even at
server side we can realloc that buffer, but before doing this let's find
a case when increasing buffer size is really useful. And let's also not
forget that making buffer too big at server with that size controllable
by connection is not good from security POV.
------------------------------------------------------------------------------
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from
the latest Intel processors and coprocessors. See abstracts and register >
http://pubads.g.doubleclick.net/gampad/clk?id=60135031&iu=/4140/ostg.clktrk
Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel

[Alex Peshkoff]

Post by marius adrian popa
Isn't the recent kernels do autotuning for tcp buffer sizes ?
http://fasterdata.es.net/host-tuning/background/#t1

It's not related to our internal buffer size.

[Carlos H. Cantu]

I really dont remember if I was able to retest with > 32K, but I found
this message that suggests that the effect of changing this parameter
is different with FB 2.5 compared to 2.1:

--------------------- 21-Dec-2011
I ran more tests tonight, and results were very different from the
past year test.

FIRST TEST (using the same server as last year test)
==========

Last year test:
Firebird 2.1.1 (SuperServer - linux) in server and FB 2.1.x Windows
in client

Tonight test:
Firebird 2.5.1 client, and same remote server as last year

First run used TcpRemoteBufferSize = 8K (the default), and
second run used TcpRemoteBufferSize = 32767 at client. I can't change
fb.conf at this server.

No difference in the fetchall time!

SECOND TEST
===========

Another test, this time connecting to one of my customers server,
internet connection by cable modem. Server is 2.5.1 (SuperClassic
Linux) and Client is 2.5.1 (Windows). Access by OpenVPN.

As I have root access to this server, I was able to change the
value of TcpRemoteBufferSize in the server and, of course, in the
client. I tested:

Server x Client
8K 8K
8K 16K
8K 32K
32K 8K
32K 32K

The fetch times were almost identical in all the above configurations
:(

The number of records returned was about 3.500, and average time was
about 11 seconds.

I restarted the server after every change at (remote) fb.conf. I also
ran the select 2 times before getting the fetchall times, the first
run was just to fill FB and disk cache.

Is there any chance that FB 2.5.1 is ignoring this parameter? How to
explain such different behavior compared to the last year test (FB 2.1)
?

PS: Last year test I used ZeosDB app as I wish to compare MySQL to FB.
Tonight test I used tool built with "FIBPlus". Also, the database used
this time was not the same as the last year. Anyway, this doesn't
matter, since what is being questioned is the fact that this time, the
parameter changing didn't make any difference in the featchall time.
I'm also trusting that on Windows, the client library will use the
fb.conf located at the directory pointed in the FB registry key.
---------------------

[]s
Carlos
http://www.firebirdnews.org
FireBase - http://www.FireBase.com.br

Post by Alex Peshkoff
Telling true I do not remember it...
But quite possible.

DS> Most likely my memory doesn't serve me well. Google shows me only thread where Carlos
DS> H. Cantu tested it up to current limit and got 3x speed up, then DE suggested to test it
DS> beyond the limit and then all was over.

[Alex Peshkoff]

Post by Carlos H. Cantu
Is there any chance that FB 2.5.1 is ignoring this parameter?

"Any" chance is always present, but looking at the code I can hardly
imagine that parameter is ignored.

Post by Carlos H. Cantu
How to
explain such different behavior compared to the last year test (FB 2.1)
?
PS: Last year test I used ZeosDB app as I wish to compare MySQL to FB.
Tonight test I used tool built with "FIBPlus". Also, the database used
this time was not the same as the last year. Anyway, this doesn't
matter, since what is being questioned is the fact that this time, the
parameter changing didn't make any difference in the featchall time.
I'm also trusting that on Windows, the client library will use the
fb.conf located at the directory pointed in the FB registry key.

Last is very easy to check. Set invalid RemoteServicePort - you should
be unable to connect.

[Alex Peshkoff]

Post by Carlos H. Cantu
Server x Client
8K 8K
8K 16K
8K 32K
32K 8K
32K 32K
The fetch times were almost identical in all the above configurations
:(
The number of records returned was about 3.500, and average time was
about 11 seconds.
I restarted the server after every change at (remote) fb.conf. I also
ran the select 2 times before getting the fetchall times, the first
run was just to fill FB and disk cache.
Is there any chance that FB 2.5.1 is ignoring this parameter? How to
explain such different behavior compared to the last year test (FB 2.1)
?

I've tried to test fb3. I see no perf difference between 2K and 32K buffers.
I'm pretty sure buffer size was changed - I've uncommented debug logging
to make sure.

[Dimitry Sibiryakov]

Post by Alex Peshkoff
I've tried to test fb3. I see no perf difference between 2K and 32K buffers.

Was channel latency high enough?

[Alex Peshkoff]

Post by Dimitry Sibiryakov

Post by Alex Peshkoff
I've tried to test fb3. I see no perf difference between 2K and 32K buffers.

Was channel latency high enough?

No.
Very small.

[Dimitry Sibiryakov]

Post by Alex Peshkoff
No.
Very small.

Buffer size has effect on fast query + latent (but wide) channel + fast client because
it decreases number of round-trips. Otherwise network waits will be beyond recognition.