wiki:sysadminPostgreSQLConfiguration

Version 37 (modified by risard, 11 years ago) (diff)

--

Installing Postgres

Quick Reference


Overview
Installing the Software
Configuring The Database Server

Creating A Data Cluster
Initializing The Database Server
Starting the Database

Creating the Evergreen Database

Installing The Schema From Source
Restoring From A Dump File

Connecting the Database Server To The Rest Of Evergreen
References


Overview

PostgreSQL is a powerful object-relational open source database server. Evergreen uses Postgres as its backend datastore. Earlier versions of Evergreen used version 8.2, currently (post v1.4) Evergreen uses Postgresql 8.3. The instructions below are for installing 8.3. See the references below for instructions on installing version 8.2 if you're using an older version of Evergreen.

The assumption being made in this documentation, is that the database server is separate from the open-ils server.


Installing the Software

Postgres is a common default install on most distros and might already be installed. As root (using either sudo or su), run apt to see:

    sudo apt-get update
    sudo apt-get install postgresql-8.3

If postgres is already installed, apt will say so and make no changes, otherwise it will be installed with default settings.

During installation,a postgres user will be added to the local system (ie the computer). This is a standard linux user account and it's permissions are created during setup. Do not change the permissions of this user! This user runs the postgres processes and should have only the permissions necessary to do that. Also created during installation is a db role called postgres. This is internal to postgres and has nothing to do with the operating system. The postgres role provides complete access to all databases created on that db server. It's important to change the passwords for both the postgres account and postgres role for security purposes.

    sudo passwd postgres

You will be prompted to change postgres's password. After doing so, you can check that it worked by trying:

    su postgres

You should gain access to the postgres account with the password you entered. If not, you likely made a mistake and can try again with the passwd command.

You change the db role via psql. Switch users to postgres:

    su postgres

and log into psql:

    psql

Run the following:

    ALTER ROLE postgres WITH ENCRYPTED PASSWORD 'mypassword';  #include the quotes around mypassword

Configuring The Database Server

Creating A Data Cluster

Postgres uses the term "database cluster" to describe the directory it uses to store data. During the apt install, a default directory is created, usually at: /var/lib/postgresql/8.3/main.

NOTE: our servers have been partitioned to best serve the needs of Evergreen. The default location is not usually on a partition that has been sized appropriately for the Evergreen db. Generally, you will not be using the default install directory for your database cluster.

You will need to determine the appropriate partition to install the cluster on. Usually, this is /data. To be sure, you can use the disk free command to see:

    df -h

should result in something like:

    Filesystem        Size  Used Avail Use% Mounted on

    /dev/mapper/LogVol00-VolGroup03
                      4.7G  145M  4.4G   4% /usr/local
    /dev/mapper/LogVol00-VolGroup07
                      1.9G  1.1G  797M  57% /var
    /dev/mapper/LogVol00-VolGroup08
                      1.9G   53M  1.8G   3% /var/log
    /dev/mapper/LogVol00-VolGroup09
                       84G   41G   39G  52% /data

Looking at the above, clearly the /data patition is the largest (84G) compared to the others.

NOTE: Having located the largest partition, you will need to create directory for the data cluster, and give the postgres user full access to it. In the spirit of keeping things standardized, we have been using the /postgresql/<version>/main format.
    sudo mkdir /data/postgresql/8.3/main
    sudo chown postgres:postgres /data/postgresql/8.3/main

Initializing The Database Server

Now you need to initialize the data cluster. Remember to include the -D switch to point it at the new directory you created above and not the default one!

    su postgres
    initdb -D /data/postgresql/8.3/main -E UNICODE --locale=C

the -E UNICODE options tells Postgres to encode the db with Unicode, and the --locale=C option tells postgres to use ISO C for localization support.

Now some configuration is required to ensure the best possible performace parameters for Evergreen. In:

    /data/postgresql/8.3/main/postgresql.conf

set the following parameters:

    # Conifer settings
    effective_cache_size = 4GB
    #shared_buffers = 512MB
    work_mem = 256MB
    #max_fsm_pages = 250000
    max_fsm_pages = 750000
    maintenance_work_mem = 1GB
    checkpoint_segments = 64
    log_line_prefix = '<%u@%h %t>'
    log_statement = all
    autovacuum = on
    stats_row_level = on
    shared_buffers = 2048MB
    default_statistics_target = 100

Starting the Database

To start the database:

    su postgres
    /usr/lib/postgresql/8.3/bin/pg_ctl start -D /data/postgresql/8.3/main -l /<path_to_log_file>/<my_logfile> #log file is optional

Notice the -D option is again set and aimed at the right data cluster. You must use the -D option every time you start or restart the server. The log file can be anywhere the postgres user can write to. If the file doesn't exist, it will be created automatically. pg_ctl lives in /usr/lib/postgresql/8.3/bin by default. You can add it to PATH by adding the following to your shell start-up file, likely ~/.bash_profile

    PATH=/usr/local/pgsql/bin:$PATH
    export PATH

Creating the Evergreen Database

Installing The Schema From Source

All of the following commands should be executed as the postgres user, so start with:

    su postgres

First you need to create a new db user which can be used to run manage the Evergreen database:

    createuser -P -s evergreen

Next, you need to: 1) create the db, 2) add a few programming languages used by Evergreen and 3) add some contributed modules to postgresql that provide extra functionality used by Evergreen like the ability to do cross-tab queries and full text searching.

    createdb -E UNICODE evergreen
    createlang plperl evergreen
    createlang plperlu evergreen
    createlang plpgsql evergreen
    psql -f /usr/share/postgresql/8.3/contrib/tablefunc.sql evergreen
    psql -f /usr/share/postgresql/8.3/contrib/tsearch2.sql evergreen
    psql -f /usr/share/postgresql/8.3/contrib/pgxml.sql evergreen
NOTE: the location of the contrib directory will change depending on how you installed postgres. If you can't find one of the contrib modules listed above, use a find command to locate them ie find / -name pgxml.sql

Now run eg_db_config.pl on the machine where Open-ils is installed. These instructions assume a two server install - one for Open-ils and Opensrf and one for postgres. eg_db_config.pl is found on the Open-ils machine. It will likely be in Open-ILS/src/support-scripts depending on where you installed the Open-ils source. If you can't find it, run "find" on eg_db_config.pl when you find it run:

    perl Open-ILS/src/support-scripts/eg_db_config.pl --create-schema
NOTE: this script uses opensrf.xml to figure out where the db server is, so it's important to have already configured opensrf to run Evergreen before running this script

You should now have a clean install of the Evergreen schema loaded with the "example consortium."

NOTE: after making any changes to the database or schema, you should run autogen.sh on the oils server to update opac and Staff Client files. Do this on the oils server as the opensrf user with oils up and running

Restoring From A Dump File

It Open-ils is running stop it first.

Then, doing the following as the postgres user:

    su postgres

In psql:

    psql 
    psql# drop database evergreen;

Note I don't use psql -d evergreen, you can't drop the db if you're using it. Check to see that there's an evergreen user before rebuilding the server. In psql, display the users:

    psql# \du

if you don't see an evergreen user:

                                   List of roles
     Role name | Superuser | Create role | Create DB | Connections | Member of
    -----------+-----------+-------------+-----------+-------------+-----------
    evergreen  | yes       | yes         | yes       | no limit    | {}
    postgres   | yes       | yes         | yes       | no limit    | {}
(2 rows)

then create the evergreen user with:

    createuser -P -s evergreen

Next, recreate the database:

    createdb -E UNICODE evergreen

Edit postgresql.conf in the data cluster (probably in /data/postgresql/8.3/main) and change:

    checkpoint_segments = 128

This is necessary to prevent errors during the restore due to the huge side of the database.

To start the restore, run:

    pg_restore -Fc -d evergreen /<path_to_my_dump_file>/<dump_file>
NOTE: It's important not to run eg_db_config.pl if you use the -d option. This seems to cause conflicts during restore

Finally, after the restore is finished (which will take 6 - 8 hours likely) run, as postgres:

    su postgres
    psql -d evergreen
    psql# VACUUM
    psql# ANALYZE

Test that everything is there by running a few queries.

NOTE: after making any changes to the database or schema, you should run autogen.sh on the oils server to update opac and Staff Client files. Do this on the oils server as the opensrf user with oils up and running

Connecting the Database Server To The Rest Of Evergreen

For a complete discussion of iptables see iptables and network configuration. For the specific ports used by the postgres server see Necessary Ports - Database Server for other postgres specific networking issues, see Special Postgresql Issues.

NOTE: The Postgres server needs JSON::XS perl mod installed in order to be able to communicate with the oils server. See Special Postgresql Issues for more information.

References

http://evergreen-ils.org/dokuwiki/doku.php?id=server:1.4.0:install
http://evergreen-ils.org/dokuwiki/doku.php?id=postgresql:8.2:debian
http://developer.postgresql.org/pgdocs/postgres/auth-pg-hba-conf.html
http://coffeecode.net/archives/156-Tuning-PostgreSQL-for-Evergreen-on-a-test-server.html
http://www.postgresql.org/docs/8.3/static/index.html
http://www.stuartellis.eu/articles/postgresql-setup/


Quick Reference

Install

    sudo apt-get update
    sudo apt-get install postgresql-8.3

#change key passwords

    sudo passwd postgres    #Linux user account
    psql# ALTER ROLE postgres WITH ENCRYPTED PASSWORD 'mypassword';  #include the quotes around mypassword

#Configuring The Database Server 
#Creating A Data Cluster on data partition

    sudo mkdir /data/postgresql/8.3/main
    sudo chown postgres:postgres /data/postgresql/8.3/main

#Initializing The Database Server - using the new cluster
    
    su postgres
    initdb -D /data/postgresql/8.3/main -E UNICODE --locale=C

#now configure postgresql.conf as per the file snippet at the end of these instructions
#start the database:
    
    su postgres
    /usr/lib/postgresql/8.3/bin/pg_ctl start -D /data/postgresql/8.3/main -l /<path_to_log_file>/<my_logfile> #log file is optional

Create the Evergreen Database

FROM SOURCE

    su postgres
    createuser -P -s evergreen
    createdb -E UNICODE evergreen
    createlang plperl evergreen
    createlang plperlu evergreen
    createlang plpgsql evergreen
    psql -f /usr/share/postgresql/8.3/contrib/tablefunc.sql evergreen
    psql -f /usr/share/postgresql/8.3/contrib/tsearch2.sql evergreen
    psql -f /usr/share/postgresql/8.3/contrib/pgxml.sql evergreen
    perl Open-ILS/src/support-scripts/eg_db_config.pl --create-schema #run on the oils server with oils running

From a dump file

#stop oils
#drop evergreen db

    su postgres
    psql# drop database evergreen;
    createuser -P -s evergreen
    createdb -E UNICODE evergreen

#edit postgresql.conf checkpoint_segments (see file snippet)
#start the restore

    pg_restore -Fc -d evergreen /<path_to_my_dump_file>/<dump_file>

#when finished clean up

    su postgres
    psql -d evergreen
    psql# VACUUM
    psql# ANALYZE

Install JSON::XS

    which gcc

#if not installed:

    apt-get update
    apt-get install gcc
    perl -MCPAN -e 'CPAN::Shell?install(“JSON::XS”)'

NETWORKING ISSUES

iptables and network configuration
Necessary Ports - Database Server
Special Postgresql Issues


###############################################################
#
# File snippet:
# /data/postgresql/8.3/main/postgresql.conf
#
###############################################################


    # Conifer settings
    effective_cache_size = 4GB
    #shared_buffers = 512MB
    work_mem = 256MB
    #max_fsm_pages = 250000
    max_fsm_pages = 750000
    maintenance_work_mem = 1GB
    checkpoint_segments = 64
    log_line_prefix = '<%u@%h %t>'
    log_statement = all
    autovacuum = on
    stats_row_level = on
    shared_buffers = 2048MB
    default_statistics_target = 100

    checkpoint_segments = 128

###############################################################