Database performance problems

--_000_7965A9DCF12CC14984420BCC37B1608F257EE116DEElzargrantc ou_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Dear all,

I have been using Postgres but I am not an expert on the subject.

I would be very grateful if someone could point to the right direction with=
regards this problem.

We have a web application written in JAVA, using jaboss as a servlet.
There are 100 connections open to the database server at any given time, ap=
parently using hibernate.

We are using Postgres 8.2.4 compiled
The database size is 155GB
We have 8GB of RAM
300GB Hard disk Raid1
Everything is within a single Volume

Today the load average was as high as 15

Top result:

Tasks: 169 total, 30 running, 139 sleeping, 0 stopped, 0 zombie
Cpu(s): 69.6% us, 5.9% sy, 0.0% ni, 0.0% id, 24.3% wa, 0.2% hi, 0.0% s=
i
Mem: 8251404k total, 8228428k used, 22976k free, 16296k buffers
Swap: 1052248k total, 47176k used, 1005072k free, 6664308k cached

I am quite sure:
1 - we do not have enough memory
2 - our disk and RAID array setup is not good enough
3 - Postgres is not setup correctly and uses an older version
4 - It is using Slony and replication which does not work.

I would like if anyone could help with the following:
1 - Ways of proving my thoughts
2 - Gather data evidence to prove

I don't have slow query check enabled, will do after Christmas, but how can=
I use it to show things are not good?

If anyone could help me it would be very much appreciated.

If you need more info or details, I am very happy to give more.

Thank you very much

Best regards

Renato Oliveira

Renato Oliveira

e-mail: renato.oliveira [at] grant.co.uk

Tel: +44 (0)1763 260811
Fax: +44 (0)1763 262410
www.grant.co.uk<http://www.grant.co.uk/>

Grant Instruments (Cambridge) Ltd

Company registered in England, registration number 658133

Registered office address:
29 Station Road,
Shepreth,
CAMBS SG8 6GB
UK

P Please consider the environment before printing this email
CONFIDENTIALITY: The information in this e-mail and any attachments is conf=
idential. It is intended only for the named recipients(s). If you are not t=
he named recipient please notify the sender immediately and do not disclose=
the contents to another person or take copies.

VIRUSES: The contents of this e-mail or attachment(s) may contain viruses w=
hich could damage your own computer system. Whilst Grant Instruments (Cambr=
idge) Ltd has taken every reasonable precaution to minimise this risk, we c=
annot accept liability for any damage which you sustain as a result of soft=
ware viruses. You should therefore carry out your own virus checks before o=
pening the attachment(s).

OpenXML: For information about the OpenXML file format in use within Grant =
Instruments please visit our website<http://www.grant.co.uk/Support/openxml=
..html>

--_000_7965A9DCF12CC14984420BCC37B1608F257EE116DEElzargrantc ou_
Content-Type: text/html; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<HTML xmlns=3D"http://www.w3.org/TR/REC-html40" xmlns:m=3D"http://schemas.m=
icrosoft.com/office/2004/12/omml" xmlns:o=3D"urn:schemas-microsoft-com:offi=
ce:office" xmlns:v=3D"urn:schemas-microsoft-com:vml" xmlns:w=3D"urn:schemas=
-microsoft-com:office:word"><head><META content=3D"text/html; charset=3Dus-=
ascii" http-equiv=3D"Content-Type">
<META content=3D"text/html; charset=3Dus-ascii" http-equiv=3D"Content-Type"=
>

<meta content=3D"text/html; charset=3Dus-ascii" http-equiv=3DContent-Type>
<meta content=3D"Microsoft Word 12 (filtered medium)" name=3DGenerator>
<style>

</style>

</head><BODY>
<DIV STYLE=3D"FONT-SIZE: 9pt; FONT-FAMILY: Courier New">
<DIV>
<DIV><FONT FACE=3D"Arial" SIZE=3D"2">

<div class=3DSection1>

<p class=3DMsoNormal>Dear all,<o:p></o:p></p>

<p class=3DMsoNormal><o:p> </o:p></p>

<p class=3DMsoNormal>I have been using Postgres but I am not an expert on t=
he
subject.<o:p></o:p></p>

<p class=3DMsoNormal><o:p> </o:p></p>

<p class=3DMsoNormal>I would be very grateful if someone could point to the=
right
direction with regards this problem.<o:p></o:p></p>

<p class=3DMsoNormal><o:p> </o:p></p>

<p class=3DMsoNormal>We have a web application written in JAVA, using jabos=
s as a
servlet.<o:p></o:p></p>

<p class=3DMsoNormal>There are 100 connections open to the database server =
at any
given time, apparently using hibernate.<o:p></o:p></p>

<p class=3DMsoNormal><o:p> </o:p></p>

<p class=3DMsoNormal>We are using Postgres 8.2.4 compiled<o:p></o:p></p>

<p class=3DMsoNormal>The database size is 155GB<o:p></o:p></p>

<p class=3DMsoNormal>We have 8GB of RAM<o:p></o:p></p>

<p class=3DMsoNormal>300GB Hard disk Raid1 <o:p></o:p></p>

<p class=3DMsoNormal>Everything is within a single Volume<o:p></o:p></p>

<p class=3DMsoNormal><o:p> </o:p></p>

<p class=3DMsoNormal>Today the load average was as high as 15<o:p></o:p></p=
>

<p class=3DMsoNormal><o:p> </o:p></p>

<p class=3DMsoNormal>Top result:<o:p></o:p></p>

<p class=3DMsoNormal><o:p> </o:p></p>

<p class=3DMsoNormal>Tasks: 169 total, 30 running, 139
sleeping,   0 stopped,   0 zombie<o:p></o:p></p>

<p class=3DMsoNormal>Cpu(s): 69.6% us, 5.9% sy, 0.0% ni, =
0.0%
id, 24.3% wa, 0.2% hi, 0.0% si<o:p></o:p></p>

<p class=3DMsoNormal>Mem:   8251404k total, 8228428k used,&=
nbsp;
22976k free,    16296k buffers<o:p></o:p></p>

<p class=3DMsoNormal>Swap: 1052248k total,    47176k
used, 1005072k free, 6664308k cached<o:p></o:p></p>

<p class=3DMsoNormal><o:p> </o:p></p>

<p class=3DMsoNormal>I am quite sure:<o:p></o:p></p>

<p class=3DMsoNormal>1 – we do not have enough memory<o:p></o:p></p>

<p class=3DMsoNormal>2 – our disk and RAID array setup is not good en=
ough<o:p></o:p></p>

<p class=3DMsoNormal>3 – Postgres is not setup correctly and uses an =
older
version<o:p></o:p></p>

<p class=3DMsoNormal>4 – It is using Slony and replication which does=
not
work.<o:p></o:p></p>

<p class=3DMsoNormal><o:p> </o:p></p>

<p class=3DMsoNormal>I would like if anyone could help with the following:<=
o:p></o:p></p>

<p class=3DMsoNormal>1 – Ways of proving my thoughts<o:p></o:p></p>

<p class=3DMsoNormal>2 – Gather data evidence to prove<o:p></o:p></p>

<p class=3DMsoNormal><o:p> </o:p></p>

<p class=3DMsoNormal>I don’t have slow query check enabled, will do a=
fter
Christmas, but how can I use it to show things are not good?<o:p></o:p></p>

<p class=3DMsoNormal><o:p> </o:p></p>

<p class=3DMsoNormal>If anyone could help me it would be very much apprecia=
ted.<o:p></o:p></p>

<p class=3DMsoNormal><o:p> </o:p></p>

<p class=3DMsoNormal>If you need more info or details, I am very happy to g=
ive
more.<o:p></o:p></p>

<p class=3DMsoNormal><o:p> </o:p></p>

<p class=3DMsoNormal>Thank you very much<o:p></o:p></p>

<p class=3DMsoNormal><o:p> </o:p></p>

<p class=3DMsoNormal>Best regards<o:p></o:p></p>

<p class=3DMsoNormal><o:p> </o:p></p>

<p class=3DMsoNormal>Renato Oliveira<o:p></o:p></p>

<p class=3DMsoNormal><o:p> </o:p></p>

</div>

</FONT></DIV>
<DIV><FONT FACE=3D"Arial" SIZE=3D"2"></FONT> </DIV>
<DIV><FONT FACE=3D"Arial" SIZE=3D"2"><FONT FACE=3D"Arial" SIZE=3D"2">Renato=
Oliveira<BR><BR>e-mail: renato.oliveira [at] grant.co.uk</FONT></FONT><FONT FAC=
E=3D"Arial" SIZE=3D"2"><FONT FACE=3D"Arial" SIZE=3D"2"></FONT></FONT></DIV>
<DIV><FONT FACE=3D"Arial" SIZE=3D"2"><FONT FACE=3D"Arial" SIZE=3D"2"></FONT=
></FONT> </DIV>
<DIV><FONT FACE=3D"Arial" SIZE=3D"2"><FONT FACE=3D"Arial" SIZE=3D"2">Tel: +=
44 (0)1763 260811<BR>Fax: +44 (0)1763 262410<BR><A HREF=3D"http://www.grant=
..co.uk/">www.grant.co.uk</A></FONT></FONT></DIV>
<DIV><FONT FACE=3D"Arial" SIZE=3D"2"><FONT FACE=3D"Arial" SIZE=3D"2"></FONT=
></FONT> </DIV>
<DIV><FONT FACE=3D"Arial" SIZE=3D"2"><FONT FACE=3D"Arial" SIZE=3D"2">Grant =
Instruments (Cambridge) Ltd <BR> <BR>Company registered in England, re=
gistration number 658133<BR> <BR>Registered office address:<BR>29 Stat=
ion Road, <BR>Shepreth, <BR>CAMBS SG8 6GB <BR>UK</FONT></FONT></DIV>
<DIV><FONT FACE=3D"Arial" SIZE=3D"2"><FONT FACE=3D"Arial" SIZE=3D"2"></FONT=
></FONT><FONT FACE=3D"Arial" SIZE=3D"2"><FONT FACE=3D"Arial" SIZE=3D"2"></F=
ONT></FONT> </DIV>
<DIV><FONT FACE=3D"Arial" SIZE=3D"2"><FONT FACE=3D"Arial" SIZE=3D"2"></FONT=
></FONT></DIV>
<DIV><FONT FACE=3D"Arial" SIZE=3D"2"><FONT FACE=3D"Arial" SIZE=3D"2"></FONT=
></FONT> </DIV>
<DIV><FONT FACE=3D"Arial" SIZE=3D"2"><FONT FACE=3D"Arial" SIZE=3D"2"></FONT=
></FONT></DIV>
<DIV> </DIV>
<DIV><FONT FACE=3D"Arial" SIZE=3D"2"><FONT FACE=3D"Arial" SIZE=3D"2"></FONT=
></FONT></DIV>
<DIV> </DIV> </DIV>
<DIV>
<P CLASS=3D"MsoNormal"><EM><B><SPAN LANG=3D"EN-US" STYLE=3D"FONT-SIZE: 18pt=
; COLOR: green; FONT-FAMILY: Webdings"></SPAN></B></EM> </P>
<P CLASS=3D"MsoNormal"><EM><B><SPAN LANG=3D"EN-US" STYLE=3D"FONT-SIZE: 18pt=
; COLOR: green; FONT-FAMILY: Webdings">P</SPAN></B></EM><EM><B><SPAN LANG=
=3D"EN-US" STYLE=3D"FONT-SIZE: 10pt; COLOR: black; FONT-FAMILY: 'Verdana','=
sans-serif'"> </SPAN></B></EM><STRONG><I><SPAN STYLE=3D"FONT-SIZE: 7.5pt; C=
OLOR: green; FONT-FAMILY: 'Arial','sans-serif'">Please consider the environ=
ment before printing this email</SPAN></I></STRONG></P></DIV>
<DIV><FONT FACE=3D"Arial" SIZE=3D"2"><STRONG>CONFIDENTIALITY</STRONG>: The =
information in this e-mail and any attachments is confidential. It is inten=
ded only for the named recipients(s). If you are not the named recipient pl=
ease notify the sender immediately and do not disclose the contents to anot=
her person or take copies. </FONT></DIV>
<DIV><FONT FACE=3D"Arial" SIZE=3D"2"></FONT> </DIV>
<DIV><FONT FACE=3D"Arial" SIZE=3D"2"><STRONG></STRONG></FONT></DIV>
<DIV><FONT FACE=3D"Arial" SIZE=3D"2"><STRONG>VIRUSES:</STRONG> The contents=
of this e-mail or attachment(s) may contain viruses which could damage you=
r own computer system. Whilst Grant Instruments (Cambridge) Ltd has taken e=
very reasonable precaution to minimise this risk, we cannot accept liabilit=
y for any damage which you sustain as a result of software viruses. You sho=
uld therefore carry out your own virus checks before opening the attachment=
(s).</FONT></DIV>
<DIV> </DIV>
<DIV><FONT FACE=3D"Arial" SIZE=3D"2"></FONT></DIV>
<DIV><FONT FACE=3D"Arial" SIZE=3D"2"><STRONG>OpenXML</STRONG>: For informat=
ion about the OpenXML file format in use within Grant Instruments please vi=
sit our <A HREF=3D"http://www.grant.co.uk/Support/openxml.html">website</A>=
</FONT></DIV></DIV></BODY></HTML>

--_000_7965A9DCF12CC14984420BCC37B1608F257EE116DEElzargrantc ou_--

Renato Oliveira [ Mi, 23 Dezember 2009 15:45 ] [ ID #2026882 ]

Re: Database performance problems

On Wed, 2009-12-23 at 14:45 +0000, Renato Oliveira wrote:
> Dear all,
>
>
>
> I have been using Postgres but I am not an expert on the subject.
>
>
>
> I would be very grateful if someone could point to the right direction
> with regards this problem.
>
>
>
> We have a web application written in JAVA, using jaboss as a servlet.
>
> There are 100 connections open to the database server at any given
> time, apparently using hibernate.
>
>
>
> We are using Postgres 8.2.4 compiled
>
> The database size is 155GB
>
> We have 8GB of RAM
>
> 300GB Hard disk Raid1
>
> Everything is within a single Volume
>
>
>
> Today the load average was as high as 15

Linux?
Are you using elevator=deadline?
How many spindles, what type of raid?
Are you doing proper maintenance?

Joshua D. Drake

--
PostgreSQL.org Major Contributor
Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564
Consulting, Training, Support, Custom Development, Engineering
Respect is earned, not gained through arbitrary and repetitive use or Mr. or Sir.

--
Sent via pgsql-admin mailing list (pgsql-admin [at] postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Joshua Drake [ Mi, 23 Dezember 2009 18:31 ] [ ID #2026883 ]

Re: Database performance problems

Renato Oliveira <renato.oliveira [at] grant.co.uk> wrote:

> We have a web application written in JAVA, using jaboss as a
> servlet.

Is the web app on the same box as the database, or separate?

> There are 100 connections open to the database server at any given
> time

How many are typically active at peak times? You could run
something like this as a database superuser at peak times a few
times to get a feel for it:

select count(*) from pg_stat_activity
where current_query <> '<IDLE>';

> We are using Postgres 8.2.4 compiled

You're missing two years and eight months of fixes for 8.2.

http://www.postgresql.org/support/versioning

> The database size is 155GB

Any idea how much of that is "active" -- in the sense of being
frequently referenced versus more-or-less archival?

> 300GB Hard disk Raid1

Two spindles is rather small for a database of that size.

> Today the load average was as high as 15

On how many CPUs?

> Top result:

It would be more useful to run vmstat 1 or maybe even iostat 1 at
peak times and capture a view of activity over time. The memory
information from top isn't always that reliable.

> 1 - we do not have enough memory

More RAM would probably improve performance; hard to tell by how
much without more information.

> 2 - our disk and RAID array setup is not good enough

More spindles would probably improve performance; hard to tell by
how much without more information.

> 3 - Postgres is not setup correctly and uses an older version

Version upgrade would almost certainly help. For configuration,
could you strip all comments and blank lines from your
postgresql.conf file and show it?

> 4 - It is using Slony and replication which does not work.

Sorry, I don't know Slony....

-Kevin

--
Sent via pgsql-admin mailing list (pgsql-admin [at] postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Kevin Grittner [ Mo, 28 Dezember 2009 19:56 ] [ ID #2027249 ]

Re: Database performance problems

On Wed, Dec 23, 2009 at 7:45 AM, Renato Oliveira
<renato.oliveira [at] grant.co.uk> wrote:
> There are 100 connections open to the database server at any given time,
> apparently using hibernate.
>
> We are using Postgres 8.2.4 compiled

Update to 8.2.latest at your earliest chance.

> The database size is 155GB
> We have 8GB of RAM
> 300GB Hard disk Raid1
> Everything is within a single Volume
> Today the load average was as high as 15
>
> Top result:
> Tasks: 169 total,=A0 30 running, 139 sleeping,=A0=A0 0 stopped,=A0=A0 0 z=
ombie
> Cpu(s): 69.6% us,=A0 5.9% sy,=A0 0.0% ni,=A0 0.0% id, 24.3% wa,=A0 0.2% h=
i,=A0 0.0% si
> Mem:=A0=A0 8251404k total,=A0 8228428k used,=A0=A0=A0 22976k free,=A0=A0=
=A0 16296k buffers
> Swap:=A0 1052248k total,=A0=A0=A0 47176k used,=A0 1005072k free,=A0 66643=
08k cached

So, 6.6G of kernel cache, so you're not starving your machine of memory.

24.3% wait means that you've got 1 out of 4 cores waiting on IO all
the time (assuming you've got a quad core machine here)

> I am quite sure:
> 1 =96 we do not have enough memory

Given how much is being used for kernel cache you're probably ok. You
could likely increase shared_buffers and work_mem and
maintenance_work_mem a bit each and use some more for the db instead
of letting the kernel have it all. Generally 1/4 mem to
shared_buffers is reasonable on smaller memory machines.

> 2 =96 our disk and RAID array setup is not good enough

Hard to be sure. What do

iostat -x 60
AND
vmstat 60

say after running for a few minutes?

> 3 =96 Postgres is not setup correctly and uses an older version

Not real old, but it is not up to date on security / bug fixes.

> 4 =96 It is using Slony and replication which does not work.

Slony is a rather complex piece of software. If you don't need it's
extra features and such, londiste from skype may be a better choice.

> I would like if anyone could help with the following:
>
> 1 =96 Ways of proving my thoughts
> 2 =96 Gather data evidence to prove

vmstat, iostat, explain analyze <yourqueryhere>

--
Sent via pgsql-admin mailing list (pgsql-admin [at] postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Scott Marlowe [ Mo, 28 Dezember 2009 21:50 ] [ ID #2027251 ]