Upgrade Instructions

Home
Upgrade Instructions

Upgrade Instructions

This page gives a detailed, step by step instructions how to do the upgrade with less pain.

Generic notes

It’s strongly recommended to backup your configs including piler.key, piler.conf, piler.pem, sphinx.conf and config-site.php before the upgrade.

If you upgrade from older releases then you should apply all the intermediate changes, ie. when upgrading from 0.1.18 to 0.1.20 please apply both db-upgrade-0.18-vs-0.19.sql and db-upgrade-0.19-vs-0.20.sql sql scripts.

Upgrading the GUI

To upgrade the GUI just overwrite everything with webui/*. It won’t touch your local configuration in config-site.php.

from 0.1.18 to 0.1.19

Execute the following sql script on the piler database:

mysql -u piler -p piler < util/db-upgrade-0.18-vs-0.19.sql

Add the following variables to config.php:

define('TABLE_GROUP', 'group');
define('TABLE_GROUP_USER', 'group_user');
define('TABLE_GROUP_EMAIL', 'group_email');

from 0.1.19 to 0.1.20

Execute the following sql script on the piler database:

mysql -u piler -p piler < util/db-upgrade-0.19-vs-0.20.sql

from 0.1.20 to 0.1.21

Add the following variables to config.php:

define('ENABLE_FOLDER_RESTRICTIONS', 0);
define('ICON_NOTES', '/view/theme/default/images/notes.png');
define('ICON_PLUS', '/view/theme/default/images/plus.gif');
define('ICON_MINUS', '/view/theme/default/images/minus.gif');
define('ICON_EMPTY', '/view/theme/default/images/1x1.gif');
define('TABLE_FOLDER', 'folder');
define('TABLE_FOLDER_USER', 'folder_user');
define('TABLE_FOLDER_EXTRA', 'folder_extra');
define('TABLE_FOLDER_MESSAGE', 'folder_message');
define('TABLE_NOTE', '`note`');
define('SPHINX_NOTE_INDEX', 'note1');
define('LOAD_SAVED_SEARCH_URL', SITE_URL . 'index.php?route=search/load');
define('MESSAGE_NOTE_URL', SITE_URL . 'index.php?route=message/note');

Execute the following sql script on the piler database:

NOTE: before running the script, you should backup the ‘tag’ and ‘note’ tables then drop them, since the new version has a modified scheme for these tables.

mysql -u piler -p piler < util/db-upgrade-0.20-vs-0.21.sql

NOTE: the built-in admin@local account cannot see the users’ emails any more. Instead the auditor role gives you the permissions to search in any mailbox. See administering piler for more.

from 0.1.21 to 0.1.22

Execute the following sql script on the piler database:

mysql -u piler -p piler < util/db-upgrade-0.21-vs-0.22.sql

Rename LANG to DEFAULT_LANG in config.php:

-define('LANG', 'en');
+define('DEFAULT_LANG', 'en');

+define('DECRYPT_ATTACHMENT_BINARY', '/usr/local/bin/pileraget');
+define('ACTION_DOWNLOAD_ATTACHMENT', 15);
+define('ACTION_UNAUTHORIZED_DOWNLOAD_ATTACHMENT', 16);
+define('SEARCH_RESULT_CHECKBOX_CHECKED', 1);
+define('REWRITE_MESSAGE_ID', 0);
+define('ENABLE_SYSLOG', 0);
+define('RESTRICTED_AUDITOR', 0);

Stop searchd, then initialize the new index called note1, and finally start searchd again:

# /etc/init.d/rc.search stop
# su - piler
$ indexer note1
$ exit
# /etc/init.d/rc.search start

If you upgraded sphinx.conf as well, then be sure to remove the following lines from sphinx.conf:

sql_attr_uint = folder

and remove the ‘folder’ column from the sql queries in the ‘main’ and ‘delta’ sources.

from 0.1.22 to 0.1.23

Execute the following sql script on the piler database:

mysql -u piler -p piler < util/db-upgrade-0.1.22-vs-0.1.23.sql

Rename config.php to config-old.php, then copy the default config.php over your config.php. From now on, the variables are in an array with their default values. If you want to override any variable, then place it to config-site.php with the new value, eg.

$config['SITE_NAME'] = 'archive.example.com';

An example config-site.php

To support new versions of piler, you MUST relocate your store directory one level down:

Old store directory (for build ⇐ 765):

$localstatedir/piler/store/

New store directory (from build >= 767):

$localstatedir/piler/store/<SERVER_ID>

By default server_id is 0 which translates to SERVER_ID: “00”. if you set server_id=14 in piler.conf, then it translates to SERVER_ID: “0e”. If you have a single piler node, then leave the default (0). The following example assumes this scenario.

Create the server_id top level store directory:

/etc/init.d/rc.piler stop

download, compile, and upgrade the new version

cd /var/piler/store
mkdir aaa
mv * aaa
mkdir 00
chown piler:piler 00
mv aaa/* 00
rmdir aaa
/etc/init.d/rc.piler start

from 0.1.23 to 0.1.24

Execute the following sql script on the piler database:

mysql -u piler -p piler < util/db-upgrade-0.1.23-vs-0.1.24.sql

Important! The index scheme has changed from a delta - main scheme to a delta-delta - delta - main scheme. However you DON’T HAVE TO change sphinx config nor the indexer related cron jobs, it just works fine. In this case copy your current $config[‘SPHINX_MAIN_INDEX’] setting to config-site.php.

If you want to upgrade the sphinx scheme, then do the following:

Stop searchd:

/etc/init.d/rc.searchd stop

comment out the current indexer cron job:

####*/15 * * * * /usr/local/bin/indexer --quiet delta1 --rotate && sleep 2 && /usr/local/bin/indexer --quiet --merge main1 delta1 --merge-dst-range deleted 0 0 --rotate

su to user piler, then execute the following (make sure the cron job indexer won’t interfere):

indexer delta1
indexer --merge main1 delta1 --merge-dst-range deleted 0 0

Overwrite sphinx.conf with the shipped version in piler-0.1.24/etc/sphinx.conf (make sure you have the proper username, password in the “source base” section.

Initialize the new index (as user piler):

indexer dailydelta

Add the new indexer cron jobs:

5,35 * * * * /usr/local/libexec/piler/indexer.delta.sh
30   2 * * * /usr/local/libexec/piler/indexer.main.sh

Start searchd:

/etc/init.d/rc.searchd start

from 0.1.24 to 1.1.0

Execute the following sql script on the piler database:

mysql -u piler -p piler < util/db-upgrade-0.1.24-vs-1.1.0.sql

from 1.1.1 to 1.2.0

There’re some minor incompatibilities from 1.1.x to 1.2.0 you must be aware of. Be sure to read RELEASE_NOTES file in the source directory describing the changes.

The most important change is that I’ve moved all piler related configs to ${sysconfdir}/piler directory (with the default options it’s /usr/local/etc/piler).

It means that whatever (piler related stuff) you had in /usr/local/etc, that must be moved to /usr/local/etc/piler, eg. /usr/local/etc/piler.conf → /usr/local/etc/piler/piler.conf, etc.

I’ve decided to put the sphinx config file to ${sysconfdir}/piler. Debian and Ubuntu ship a sphinx package which enabled a periodic indexer –all cron job, which practically destroys the sphinx indices, and despite both the install docs and the FAQ warn about it, many piler users fell for this debian ‘trick’.

To match the new path, I’ve updated the rc.searchd file, and the indexer shell scripts as well.

Execute the following sql script on the piler database:

mysql -u piler -p piler < util/db-upgrade-1.1.0-vs-1.2.0.sql

I recommend you to run pilerconf after the upgrade, and check if you get the values in piler.conf back. If so, then the config files are at the proper new location.

from 1.2.0 to 1.3.0

Nothing extra is required.

from 1.3.0 to 1.3.5

The default mysql encoding has been changed from utf8 to utf8mb4. In mysql’s terms ‘utf8’ is only a subset of the 4 byte range of the UTF-8 encoding. To provide full UTF-8 support mysql offers utf8mb4 encoding which is the default starting from 1.3.2. For new installations I recommend you to use the default (utf8mb4).

When upgrading you may either convert your database and tables to utf8mb4 (see the discussion at https://bitbucket.org/jsuto/piler/issues/709/mysql_stmt_execute-error-incorrect-string and/or google the exact steps), or keep the current mysql settings.

In the latter case be sure to set the following values:

piler.conf:

mysqlcharset=utf8

config-site.php:

$config['DB_CHARSET'] = 'utf8';

sphinx.conf (for all sql_query_pre settings!):

sql_query_pre = SET NAMES utf8

Perform the following sql statements:

alter table sph_index change column `from` `from` tinyblob default null;
alter table metadata change column `from` `from` varchar(255) not null;
alter table metadata change column `fromdomain` `fromdomain` varchar(255) not null;
alter table metadata change column `message_id` `message_id` varchar(255) not null;
alter table attachment change column `name` `name` tinyblob default null;
alter table sph_index change column `body` `body` mediumblob default null;
alter table sph_index change column `subject` `subject` blob default null;

And add the following jobs to piler’s crontab:

*/5 * * * * /usr/bin/find /var/piler/error -type f|wc -l > /var/piler/stat/error
3 * * * * /usr/local/libexec/piler/watch_sphinx_main_index.sh

from 1.4.5 to 1.4.6 in docker deployment

Due to a change in the underlying image (moving from ubuntu 22.04 LTS to 24.04 LTS) the uid/gid of the piler user changed from 1000 to 1001. So effectively all docker volumes need their file system owners updated manually:

chown -R 1001:1001 <volume_folder>

Further more the index for the auditor table was changed from sql to manticore. So the following section must be added to /etc/piler/manticore.conf if not present:

index audit1
{
    type = rt
    path = /var/piler/manticore/audit1
    rt_mem_limit = 16M
    stored_fields = *
    min_word_len = 2
    min_prefix_len = 5
    charset_table  = <?php print CHARSET_TABLE; ?>
    rt_field = email
    rt_field = ipaddr
    rt_field = description
    rt_attr_bigint = ts
    rt_attr_bigint = meta_id
    rt_attr_uint = action
}

If you have any local copies of /etc/piler/config-site.php then don’t forget to add $config['SPHINX_AUDIT_INDEX'] = 'audit1'; to reflect the change.

Last but not least the new docker image comes with php 8.3 while 8.1 was used in piler 1.4.5 image. So make sure to configure /etc/piler/piler-nginx.conf with the correct php version:

            ...
            fastcgi_pass unix:/var/run/php/php8.3-fpm.sock;
            ...

from 1.4.6 to 1.4.7 in docker deployment

The manticore index is moved to a seperate container. So you need to adjust your docker-compose.yaml file:

Add a new service manticore:

  manticore:
    image: manticoresearch/manticore:6.3.2
    container_name: manticore
    restart: unless-stopped
    volumes:
      - ./manticore.conf:/etc/manticoresearch/manticore.conf
      - piler_manticore:/var/lib/manticore
    deploy:
      resources:
        reservations:
          memory: 512M
        limits:
          memory: 512M

Create a file ./manticore.conf in your project directory and use the content from here: https://github.com/jsuto/piler/blob/piler-1.4.7/docker/manticore.conf

Adjust the service piler in your docker-compose.yaml and add the environment variable MANTICORE_HOSTNAME=manticore and remove the volume piler_manticore as this exact volume will be used with the manticore service from now on.

In order for the new docker container to pick up the existing data it’s unix owner/group must be adjusted:

chown -R 999:999 <your_piler_manticore_volume>

Also be sure to update /etc/piler/piler.conf within the piler container, and set

sphxhost=manticore

Make sure your /etc/piler/config-site.php contains the following lines:

$config['SPHINX_HOSTNAME'] = 'manticore:9306';
$config['SPHINX_HOSTNAME_READONLY'] = 'manticore:9307'

Search results: {{ num_search_results }}