---

Welcome to the Full Page Write Improvement Project Home Page

---

This project aims to reduce the size of archive log files by replacing backup blocks (full page writes) marked as "removable" (introduced in 8.3) with corresponding incremental logs.

The Full Page Write Improvementproject is a PostgreSQL Community project that is a part of the pgFoundry. It is produced by NTT OSS Center.

The pgFoundry page for the project is at http://pgfoundry.org/projects/pglesslog, where you can find downloads, documentation, bug reports, mailing lists, and a whole lot more.

日本語ページはこちら

What is lesslog?

Lesslog is a set of tools to reduce the size of PostgreSQL archive log. Lesslog consists of the following programs.

pg_compresslog

This program is used to reduce the size of WAL files when WAL is archived. This is done by replacing full page writes (physical log) of WAL records with corresponding incremental logs. Such replacement is done only for WAL records with "removable" mark. Removable mask is described in xlog.c code. Because incremental log is extremely smaller than full page writes, this program reduces WAL file size when archived. This command should be specified as archive_command in postgresql.conf. This command also removes page headers by changing page size from 8kB to 16MB, which are restored by pg_decompresslog.

pg_decompresslog

This command restores page headers and add dummy data to make up for physical log record, finally restores LSN of each log record and restores the page size to be used in the archive recovery. This command should be specified as restore_command in recovery.conf. Size of the file will be the same as original WAL segment files before archived. "NOOP" WAL record will be used for the filler, which will be neglected by redo functions at the recovery.

How to install lesslog

Environment

Pglesslog works with PostgreSQL 8.3.x. It uses "removable" flag in WAL records intruduced in PostgreSQL 8.3.

Installation procedure

• Build and install the additional tools. Move to contrib/lesslog directory, then make and make install. pg_compresslog and pg_decompresslog will be installed to PostgreSQL install directory.
• Edit postgresql.conf Edit postgresql.conf which is copied to DB cluster by initdb and edit parameters as follows.

        full_page_writes = on
        archive_command = 'pg_compresslog "%p" /"%f"'

How to use pg_compresslog

Synopsis

    pg_compresslog [from [to]]

Explanation

pg_compresslog removes physical log from the WAL segment file specified by < from > and archives as < to > file name.
if < from > is omitted or specfied as "-", WAL segment file is read from stdin. If < to > is omitted or specified as "-", archive file will be written to stdout.
Full page writes replaced with corresponding incremental logs by pg_compresslog are those written while online backup is not running.
To use the output of pg_compresslog command in archive recovery, it must be restored using pg_decompresslog command.

Return value

pg_compresslog returns zero if no error occurs, 1 if error occurs.

How to use pg_decompresslog

Synopsis

    pg_decompresslog [from [to]]

Explanation

pg_decompresslog reads archive log file specified by < from > argument and restores an area corresponds to the removed physical log, which restores LSN of each log record, and writes them to the file specified by < to > argument.
If < from > is omitted or specified as "-", it reads from stdin. If < to > is omitted of specified as "-", it writes to stdout.
Full page writes replaced with corresponding incremental logs by pg_compresslog are those written while online backup is not running.
You can specifiy the file written by pg_compresslog as < from > argument.

Return value

It returns zero if no error occurs, 1 if error occurs.

What's new in lesslog 1.1?

Lesslog 1.1 can generate incremental log for HEAP2_CLEAN WAL records, which are created by vacuum and prune (HOT). Because full page writes of this WAL records occupies major part of WAL segments, replacing this with incremental log dramatically reduces the size of WAL when archiving.

How small archive log will be?

Environment

• Scale factor: 100
• Connection: 10
• Transactions: 360,000 per connection
• shared_buffers: 32MB
• max_fsm_pages: 204,800
• checkpoint_segments: 1,000
• checkpoint_timeout: 5min
• checkpoint_completion_target: 0.5
• autovacuum: on

Size of original/compressed WAL

• WAL segments :1,379
• Original WAL size : 22,064MB
• Compressed WAL size: 3,598MB
• Size ratio: 16% of the original

Acknowledgement

I appreciate Pavan Deolasee and Heikki Linnakangas for their stimulating comments and advise. Without their advises, the solution would have been much more complicated.