pg_dump custom format and pigz

This is a multi-part message in MIME format.

------_=_NextPart_001_01CA53DF.3048C3EC
Content-Type: text/plain;
charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Hello,

I'm using pg_dump intensively, and until now using the plaintext format
which allows me to pipe the output to pigz (http://zlib.net/pigz/)
This is the fastest way I've found to generate compressed backups.

(These backups are very rarely used)

I would like to switch to the custom format in order to take advantage
of the parallel restore enhancement, but this would force me to give
pigz up, hence loosing performances with pg_dump.

Is there a way to combine the custom format with pigz?

feature request ?

best regards,

Marc Mamin

------_=_NextPart_001_01CA53DF.3048C3EC
Content-Type: text/html;
charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; =
charset=3Dus-ascii">
<META NAME=3D"Generator" CONTENT=3D"MS Exchange Server version =
6.5.7651.59">
<TITLE>pg_dump custom format and pigz</TITLE>
</HEAD>
<BODY>

 

Hello,


I'm using pg_dump intensively, and =
until now using the plaintext format which allows me to pipe the output =
to pigz (<A HREF=3D"http://zlib.net/pigz/">http://zlib.net/pigz/</A>)

This is the fastest way I've found to =
generate compressed backups.


(These backups are very rarely =
used)


I would like to switch to the custom =
format in order to take advantage of the parallel restore enhancement, =
but this would force me to give pigz up, hence loosing performances with =
pg_dump.

Is there a way to combine the custom =
format with pigz?


feature request ?


best regards,


Marc Mamin

 
 

</BODY>
</HTML>
------_=_NextPart_001_01CA53DF.3048C3EC--

Hi Marc,

On Fri, Oct 23, 2009 at 02:48:58PM +0200, Marc Mamin wrote:

> I'm using pg_dump intensively, and until now using the plaintext format
> which allows me to pipe the output to pigz (http://zlib.net/pigz/)
> This is the fastest way I've found to generate compressed backups.
>
> (These backups are very rarely used)
>
> I would like to switch to the custom format in order to take advantage
> of the parallel restore enhancement, but this would force me to give
> pigz up, hence loosing performances with pg_dump.

> Is there a way to combine the custom format with pigz?

You might add pigz as a post-processing step and disabling compression in
pg_dump.

HTH,

Tino.

--
"What we nourish flourishes." - "Was wir nähren erblüht."

www.lichtkreis-chemnitz.de
www.tisc.de

--
Sent via pgsql-admin mailing list (pgsql-admin [at] postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Hello,

> You might add pigz as a post-processing step and disabling compression in=
pg_dump.

The problem with this solution is that it make it necessary to decompress t=
he dump entirely before using pg_restore (or did I miss a point ?)
This is an issue as our uncompressed dumps may reach 3-500 GB...

A better way would be to have a configuration option to tell which compress=
ion tool should be used internally by pg_dump.

best regards,

Marc Mamin

-----Original Message-----
From: pgsql-admin-owner [at] postgresql.org [mailto:pgsql-admin-owner [at] postgresql=
..org] On Behalf Of Tino Schwarze
Sent: Friday, October 23, 2009 3:11 PM
To: pgsql-admin [at] postgresql.org
Subject: Re: [ADMIN] pg_dump custom format and pigz

Hi Marc,

On Fri, Oct 23, 2009 at 02:48:58PM +0200, Marc Mamin wrote:

> I'm using pg_dump intensively, and until now using the plaintext
> format which allows me to pipe the output to pigz
> (http://zlib.net/pigz/) This is the fastest way I've found to generate co=
mpressed backups.
>
> (These backups are very rarely used)
>
> I would like to switch to the custom format in order to take advantage
> of the parallel restore enhancement, but this would force me to give
> pigz up, hence loosing performances with pg_dump.

> Is there a way to combine the custom format with pigz?

You might add pigz as a post-processing step and disabling compression in p=
g_dump.

HTH,

Tino.

--
"What we nourish flourishes." - "Was wir nähren erblüht."

www.lichtkreis-chemnitz.de
www.tisc.de

--
Sent via pgsql-admin mailing list (pgsql-admin [at] postgresql.org) To make chan=
ges to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

--
Sent via pgsql-admin mailing list (pgsql-admin [at] postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Hi Marc,

On Fri, Oct 23, 2009 at 03:52:16PM +0200, Marc Mamin wrote:

> > You might add pigz as a post-processing step and disabling compressio=
n in pg_dump.
>
> The problem with this solution is that it make it necessary to
> decompress the dump entirely before using pg_restore (or did I miss a
> point ?)
> This is an issue as our uncompressed dumps may reach 3-500 GB...

Right, that would not work.

> A better way would be to have a configuration option to tell which
> compression tool should be used internally by pg_dump.

Well, that would mean linking pg_dump with the library providing the
compression you want. I suppose, only data is compressed, not the whole
dump, e.g. dumping without compression then compressing is different
from dumping with compression.

Tino.

--
"What we nourish flourishes." - "Was wir nähren erblüht."

www.lichtkreis-chemnitz.de
www.tisc.de

--
Sent via pgsql-admin mailing list (pgsql-admin [at] postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

>Well, that would mean linking pg_dump with the library providing the compr=
ession you want. I suppose, only data is compressed, not the whole dump, e.=
g. dumping without compression then compressing is different from dumping w=
ith compression.

Yep,
I have no skill in C programming, but wouldn't it be a good idea to add/int=
egrate the pigz source in pg_dump ? There is a least no concern to have abo=
ut the License.
regards,

Marc

-----Original Message-----
From: pgsql-admin-owner [at] postgresql.org [mailto:pgsql-admin-owner [at] postgresql=
..org] On Behalf Of Tino Schwarze
Sent: Friday, October 23, 2009 4:25 PM
To: pgsql-admin [at] postgresql.org
Subject: Re: [ADMIN] pg_dump custom format and pigz

Hi Marc,

On Fri, Oct 23, 2009 at 03:52:16PM +0200, Marc Mamin wrote:

> > You might add pigz as a post-processing step and disabling compression =
in pg_dump.
>
> The problem with this solution is that it make it necessary to
> decompress the dump entirely before using pg_restore (or did I miss a
> point ?) This is an issue as our uncompressed dumps may reach 3-500
> GB...

Right, that would not work.

> A better way would be to have a configuration option to tell which
> compression tool should be used internally by pg_dump.

Well, that would mean linking pg_dump with the library providing the compre=
ssion you want. I suppose, only data is compressed, not the whole dump, e.g=
.. dumping without compression then compressing is different from dumping wi=
th compression.

Tino.

--
"What we nourish flourishes." - "Was wir nähren erblüht."

www.lichtkreis-chemnitz.de
www.tisc.de

--
Sent via pgsql-admin mailing list (pgsql-admin [at] postgresql.org) To make chan=
ges to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

--
Sent via pgsql-admin mailing list (pgsql-admin [at] postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

pg_dump custom format and pigz

Re: pg_dump custom format and pigz

Re: pg_dump custom format and pigz

Re: pg_dump custom format and pigz

Re: pg_dump custom format and pigz