This is the help of the Vidjil server. This help is intended for server administrators. Users should consult the web application manual Other documentation can also be found in doc/. Finally, developer documentation

Docker containers or Plain installation

There are two ways to install and run a Vidjil server:

Requirements

CPU, RAM

Minimal

vidjil-algo typically uses approx. 1.2GB of RAM to run on a 1GB .fastq and will take approx. 5+ minutes. Therefore in order to process requests from a single user with a few samples, any standard multi-core processor with 2GB RAM will be enough.

When choosing hardware for your server it is important to know the scale of usage you require. If you have many users that use the app on a daily basis, you will need to have multiple cores to ensure the worker queues don't build up. One worker will occupy one core completely when running vidjil-algo (which is currently single-threaded).

For reference, here are various setups of our public testing server https://app.vidjil.org:

2016 -- 2017 (40+ users, including 15 regular users)

since 2018 (100+ users, including 30+ regular users)

We create less workers for executing Vidjil-algo than there are (virtual) CPU availabe, keeping always one CPU core dedicated to the web server, even when the workers run at full capacity. Running other RepSeq programs through the Vidjil server may require additional CPU and RAM.

Storage

Full upload of sequences

As for many high-throughput sequencing pipeline, disk storage to store input data (.fastq, .fasta, .fastq.gz or .fasta.gz) is now the main constraint in our environment.

Depending on the sequencer, files can weigh several GB. Depending of the number of users, a full installation's total storage should thus be serveral hundred GB, or even several TB (as of the end of 2018, 4 TB for the public server). We recommend a RAID setup of at least 2x2TB to allow for user files and at least one backup.

User files (results, annotations) as well as the metadata database are quite smaller (as of the end of 2016, on the public server, 3 GB for all user files of 40+ users). Note that even when the input sequences are deleted, the server is still able to display the results of previous analyses.

Remote access on a mounted filesystem

Moreover, it is possible to access .fastq files on a mounted filesystem. See FILE_SOURCE below.

Authentication

The accounts are now local to the Vidjil server. We intend to implement some LDAP access at some point of 2020.

Network

Once installed, the server can run on a private network. However, the following network access are recommended:

Docker -- Installation

All our images are hosted on DockerHub in the vidjil/ repositories. The last images are tagged with vidjil/server:latest and vidjil/client:latest.

Individual services are started by docker-compose (https://docs.docker.com/compose/).

Before installation

Install docker-compose. See https://docs.docker.com/compose/install/#install-compose

If it doesn't exist yet, you should create a docker group. The users needing to access docker must belong to this group.

Install git. Clone the Vidjil git with git clone https://gitlab.inria.fr/vidjil/vidjil.git, and go to the directory vidjil/docker. This contains both docker-compose.yml as well as configuration files.

Docker environment

The vidjil Docker environment is managed by docker-compose, who launches the following services:

From image vidjil/client

From image vidjil/server

Network usage and SSL certificates

If you are simply using Vidjil from your computer for testing purposes you can skip the next two steps.

You will need the following step whether you are using locally or not.

Vidjil uses HTTPS by default, and will therefore require SSL certificates. You can achieve this with the following steps:

If necessary, in docker-compose.yml, update nginx.volumes, line ./vidjil-client/ssl:/etc/nginx/ssl, to set the directory with the certificates. The same can be done for the postfix container.

If you would prefer to use the vidjil over HTTP (not recommended outside of testing purposes), you can use the provided configuration files in docker/vidjil-server/conf and docker/vidjil-client/conf. You will find several files that contain "http" in their name. Simply replace the existing config files with their HTTP counter-part (for safety reasons, don't forget to make a backup of any file you replace.)

First configuration and first launch

notice : By default, Nginx HTTP server listens for incoming connection and binds on port 80 on the host, if you encounter the following message error:

ERROR: for nginx
Cannot start service nginx: driver failed programming external
connectivity on endpoint docker_nginx_1
(236d0696ed5077c002718541a9703adeee0dfac66fb880d193690de6fa5c462e):
Error starting userland proxy: listen tcp 0.0.0.0:80: bind: address already in use

You can resolve it either by changing the port used by Vidjil in the nginx.ports section of the docker-compose.yml file or by stopping the service using port 80.

Further configuration

The following configuration files are found in the vidjil/docker directory:

Here are some notable configuration changes you should consider:

Docker -- Adding external software

Some software can be added to Vidjil for pre-processing or even processing if the software outputs data compatible with the .vidjil or AIRR format. We recommend you add software by adding a volume to your docker-compose.yml. By default we add our external files to /opt/vidjil on the host machine. You can then reference the executable in vidjil-server/conf/defs.py.

When the software has compatible inputs and outputs, it will be enough to configure then the appropriate pre process or analysis config (to be documented). In some cases, using the software may require development such as wrappers. Contact us (contact@vidjil.org) to have more information and help.

Docker -- Troubleshooting

Error "Can't connect to MySQL server on 'mysql'"

The mysql container is not fully launched. This can happen especially at the first launch. You may relaunch the containers.

If restarting the containers does not resolve the issue, there are a couple of things you can look into:

Launching manually the backup

The backup should be handled by the backup container, see Making backups below. Otherwise you can use the backup.sh script by connecting to the backup or uwsgi container (for a full backup, otherwise add the -i option when running backup.sh):

cd /usr/share/vidjil/server
sh backup.sh vidjil /mnt/backup >> /var/log/cron.log 2>&1

I can't connect to the web2py administration site

The URL to this site is https://mywebsite/admin/default/. The password should be given in the docker-compose.yml file. Otherwise a random password is generated. You can still modify this password by connecting to the server (in the uwsgi container). Go in the the /usr/share/vidjil/server/web2py directory and then launch Python.

from gluon.main import save_password
save_password(PASSWORD, 443)

This password will not persist when the container will be restarted. For a persistent password, please use the environment variable.

Docker -- Updating a Docker installation

Before the update

We post news on image updates at http://gitlab.vidjil.org/tree/dev/docker/CHANGELOG. Check there whether the new image require any configuration change.

By security, we please you to always make a backup (see "Backups", below) before doing this process. It is especially important to backup the database, as the update process may transform it.

Pulling the new images

docker pull vidjil/server:latest
docker pull vidjil/client:latest

This will pull the latest version of the images. More tags are available at https://hub.docker.com/r/vidjil/server/tags/.

If you do not have access to hub.docker.com on your server, then you should pull and extract the image onto a machine that does, send it to your server with your favourite method, and finally import the image on the server.

Extract:

docker save -o <output_file> vidjil/server[:<version>] vidjil/client[:<version>]

Import:

docker load -i <input_file>

Launch the new containers

In some cases, you may need to update your docker-compose.yml file or some of the configuration files. We will describe the changes in the CHANGELOG file. The latest versions of these files are available on our Gitlab.

Once the images are pulled, you can relaunch the containers:

docker-compose down
docker-compose up

By default, all previous volumes will be reused and no data will be lost. If the database schema was updated, web2py will update it on your database. Check that the containers run well, and that you still manage to log on Vidjil and to access the database, and to see a result from a sample.

If something is not working properly, you have still the option to rollback to the previous images (for example by tagging as latest a previous image), and possibly by reusing also your last databse backup if something went wrong.

Launching a single container

When an update occurs on a single container, one may not want to relaunch all the containers, to save time. With docker-compose it is possible to do so.

Stop the desired container (for instance the client):

docker-compose stop nginx

Then launch it again

docker-compose up -d nginx

Knowing what docker image version is running

As our latest image is always tagged latest you may have troubles to know what version is currently running on your server. To determine that, you can use the digest of the image. You can view it, for example with docker image --digests vidjil/server. Then you can compare it with the digests shown on the Dockerhub page.

Plain server installation

This installation is not supported anymore. We rather advise to use the Docker containers (see above).

Requirements (for Ubuntu 16.04)

apt-get install git
apt-get install g++
apt-get install make
apt-get install unzip
apt-get install python-dev python-pip
apt-get install libyajl2 libyajl-dev
pip install unittest2
pip install unittest-xml-reporting
pip install enum34
pip install ijson cffi

Server installation and initialization

Enter in the server/ directory.

If you just want to do some tests without installing a real web server, then launch make install_web2py_standalone. In the other case, launch make install_web2py.

Detailed manual server installation and browser linking

Requirements: ssh, zip unzip, tar, openssh-server, build-essential, python, python-dev, mysql, python2.5-psycopg2, postfix, wget, python-matplotlib, python-reportlab, python-enum34, mercurial, git

If you want to run Vidjil with an Apache webserver you will also need: apache2, libapache2-mod-wsgi

Or if you want to use Nginx: nginx-full, fcgiwrap

For simplicity this guide will assume you are installing to /home/www-data

Clone https://github.com/vidjil/vidjil.git

Download and unzip web2py. Copy the contents of web2py to the server/web2py folder of you Vidjil installation (in this case /home/www-data/vidjil/server/web2py) and give ownership to www-data:

chown -R www-data:www-data /home/www-data/vidjil

If you are using apache, you can run the following commands to make sure all the apache modules you need are activated:

a2enmod ssl
a2enmod proxy
a2enmod proxy_http
a2enmod headers
a2enmod expires
a2enmod wsgi
a2enmod rewrite  # for 14.04

In order to setup the SSL encryption a key to give to apache. The safest option is to get a certicate from a trusted Certificate Authority, but for testing purposes you can generate your own:

mkdir /etc/<webserver>/ssl
openssl genrsa 1024 > /etc/<webserver>/ssl/self_signed.key
chmod 400 /etc/<webserver>/ssl/self_signed.key
openssl req -new -x509 -nodes -sha1 -days 365 -key
    /etc/<webserver>/ssl/self_signed.key > /etc/apache2/ssl/self_signed.cert
openssl x509 -noout -fingerprint -text <
    /etc/<webserver>/ssl/self_signed.cert > /etc/<webserver>/ssl/self_signed.info

\<webserver> should be replaced with the appropriate webserver name (ie. apache2 or nginx)

Given that Vidjil is a two-part application, one that serves routes from a server and one that is served statically, we need to configure the apache to do so. Therefore we tell the apache to:

you can replace your apache default config with the following (/etc/apache2/sites-available/default.conf - remember to make a backup just in case):

WSGIDaemonProcess web2py user=www-data group=www-data processes=1 threads=1

<VirtualHost *:80>

  DocumentRoot /var/www
  <Directory />
    Options FollowSymLinks
    AllowOverride None
  </Directory>

  <Directory /var/www/>
    Options Indexes FollowSymLinks MultiViews
    AllowOverride all
    Order allow,deny
    allow from all
  </Directory>

  ScriptAlias /cgi/ /usr/lib/cgi-bin/

  <Directory /usr/lib/cgi-bin/>
    Options Indexes FollowSymLinks
    Options +ExecCGI
    #AllowOverride None
    Require all granted
    AddHandler cgi-script cgi pl
  </Directory>

  <Directory /home/www-data/vidjil/browser>
    AllowOverride None
  </Directory>

  CustomLog /var/log/apache2/access.log common
  ErrorLog /var/log/apache2/error.log
</VirtualHost>


<VirtualHost *:443>
  SSLEngine on
  SSLCertificateFile /etc/apache2/ssl/self_signed.cert
  SSLCertificateKeyFile /etc/apache2/ssl/self_signed.key

  WSGIProcessGroup web2py
  WSGIScriptAlias / /home/www-data/vidjil/server/web2py/wsgihandler.py
  WSGIPassAuthorization On

  <Directory /home/www-data/vidjil/server/web2py>
    AllowOverride None
    Require all denied
    <Files wsgihandler.py>
      Require all granted
    </Files>
  </Directory>

  AliasMatch ^/([^/]+)/static/(?:_[\d]+.[\d]+.[\d]+/)?(.*) \
        /home/www-data/vidjil/server/web2py/applications/$1/static/$2

  <Directory /home/www-data/vidjil/server/web2py/applications/*/static/>
    Options -Indexes
    ExpiresActive On
    ExpiresDefault "access plus 1 hour"
    Require all granted
  </Directory>

  CustomLog /var/log/apache2/ssl-access.log common
  ErrorLog /var/log/apache2/error.log
</VirtualHost>

Now we want to activate some more apache mods:

a2ensite default                   # FOR 14.04
a2enmod cgi

Restart the server in order to make sure the config is taken into account.

And create some symlinks to avoid splitting our app:

ln -s /home/www-data/vidjil/browser /var/www/browser
ln -s /home/www-data/vidjil/browser/cgi/align.cgi /usr/lib/cgi-bin
ln -s /home/www-data/vidjil/germline /var/www/germline
ln -s /home/www-data/vidjil/data /var/www/data

If you are using Nginx, the configuration is the following:

server {
    listen 80;
    server_name \$hostname;
    return 301 https://\$hostname$request_uri;

}
server {
        listen 443 default_server ssl;
        server_name     \$hostname;
        ssl_certificate         /etc/nginx/ssl/web2py.crt;
        ssl_certificate_key     /etc/nginx/ssl/web2py.key;
        ssl_prefer_server_ciphers on;
        ssl_session_cache shared:SSL:10m;
        ssl_session_timeout 10m;
        ssl_ciphers ECDHE-RSA-AES256-SHA:DHE-RSA-AES256-SHA:DHE-DSS-AES256-SHA:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA;
        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
        keepalive_timeout    70;
        location / {
            #uwsgi_pass      127.0.0.1:9001;
            uwsgi_pass      unix:///tmp/web2py.socket;
            include         uwsgi_params;
            uwsgi_param     UWSGI_SCHEME \$scheme;
            uwsgi_param     SERVER_SOFTWARE    nginx/\$nginx_version;
            ###remove the comments to turn on if you want gzip compression of your pages
            # include /etc/nginx/conf.d/web2py/gzip.conf;
            ### end gzip section

            proxy_read_timeout 600;
            client_max_body_size 20G;
            ###
        }
        ## if you serve static files through https, copy here the section
        ## from the previous server instance to manage static files

        location /browser {
            root /home/www-data/vidjil/;
            expires 1h;

            error_page 405 = $uri;
        }

        location /germline {
            root $CWD/../;
            expires 1h;

            error_page 405 = $uri;
        }

        ###to enable correct use of response.static_version
        #location ~* ^/(\w+)/static(?:/_[\d]+\.[\d]+\.[\d]+)?/(.*)$ {
        #    alias /home/www-data/vidjil/server/web2py/applications/\$1/static/\$2;
        #    expires max;
        #}
        ###

        location ~* ^/(\w+)/static/ {
            root /home/www-data/vidjil/server/web2py/applications/;
            expires max;
            ### if you want to use pre-gzipped static files (recommended)
            ### check scripts/zip_static_files.py and remove the comments
            # include /etc/nginx/conf.d/web2py/gzip_static.conf;
            ###
        }

        client_max_body_size 20G;

        location /cgi/ {
            gzip off;
            root  /home/www-data/vidjil/browser/;
            # Fastcgi socket
            fastcgi_pass  unix:/var/run/fcgiwrap.socket;
            # Fastcgi parameters, include the standard ones
            include /etc/nginx/fastcgi_params;
            # Adjust non standard parameters (SCRIPT_FILENAME)
            fastcgi_param SCRIPT_FILENAME  \$document_root\$fastcgi_script_name;
        }

}

We also do not create symlinks since all references are managed correctly.

Now we need to configure the database connection parameters:

This tells the browser to access the server on the current domain. You may also add a variable called server_id in order to name different instances and environments; it will be displayed in the top menu.

You can now access your app. All that is left to do is click on the init database link above the login page. This creates a default admin user: plop@plop.com and password: 1234 (make sure to remove this user in your production environment) and creates the configurations you can have for files and results.

Testing the server

If you develop on the server, or just want to check if everything is ok, you should launch the server tests.

First, you should have a working fuse server by launching make launch_fuse_server (just launch it once, then it is running in the background and can be killed with make kill_fuse_server).

Then you can launch the tests with make unit.

Troubleshootings

Web2py runs but does not allow any connection

Check whether the relevant disks are properly mounted. Disks failures or other events could have triggered a read-only partition.

Jobs stay in QUEUED, workers seem to be stuck

For some reasons, that are not clear yet, it may happen that workers are not assigned any additional jobs even if they don't have any ongoing jobs.

In such a (rare) case, it may be useful to restart the workers by clicking on the reset workers link in the Vidjil administration interface. Restarting workers won't be performed if jobs are currently running or assigned.

Debugging Web2py workers

One can launch the workers by hand (see in the /etc/init script and add a -D 0 option. It prints debugging information on what the workers are doing.

The most useful information are from the TICKER worker: the one that assigns jobs to workers. So you'd better first kill all the workers and then launch one by hand to be sure that it will be the ticker.

Restarting web2py

Just touch the file /etc/uwsgi/web2py.ini.

Another of restarting it is by touching the file server/web2py/applications/vidjil/modules/defs.py. This will tell uwsgi to restart web2py (including the workers).

Restarting uwsgi

When one modifies an uwsgi config file (usually in /etc/uwsgi directory, it may be necessary to restart uwsgi so that the modifications are taken into account. This can be done using

initctl restart uwsgi-emperor

Logging database queries

MySQL

One can see some insightful SO post. To summarize, this can either be done at runtime:

SET GLOBAL log_output = "FILE";
SET GLOBAL general_log_file = "/path/to/your/logfile.log";
SET GLOBAL general_log = 'ON';

Or directly in the configuration file (less recommended):

general_log_file        = /var/log/mysql/mysql.log
general_log             = 1

In that case the server must be restarted afterwards.

Running the server in a production environment

Introduction

When manipulating a production environment it is important to take certain precautionnary mesures, in order to ensure production can either be rolled back to a previous version or simply that any encurred loss of data can be retrieved.

Web2py and Vidjil are no exception to this rule.

Making backups

The top priority is to backup files created during the analysis (either by a software or a human). Should the data be lost, valuable man-hours would be lost. In order to prevent this, we make several times a day incremental backups of the data stored on the public Vidjil servers.

This does not apply to uploaded files. We inform users that they should keep a backup of their original sequence files.

To ease the backup, the backup.sh script provides an example. It can be used through the backup container, for which you have two configuration files to update.

The docker/backup/conf/backup.cnf gives the authentication information to the database so that a backup user (read rights only required) can connect to the database.

Then the backup strategy can be configured in the docker/backup/conf/backup-cron file. The cron file states how often the backup script will be called. There are three options: backing up all results/analyses since yesterday, since the start of the month, since forever. On top of that the database is exported under two formats (CSV and SQL).

Autodelete and Permissions

Web2py has a handy feature called AutoDelete which allows the administrator to state that file reference deletions should be cascaded if no other references to the file exist. When deploying to production one needs to make sure AutoDelete is deactivated. This is the case for the default Vijdil installation (see server/web2py/applications/vidjil/models/db.py).

As a second precaution it is also wise to temporarily restrict web2py's access to referenced files.

Taking two mesures to prevent file loss might seem like overkill, but securing data is more important than the small amount of extra time spent putting these mesures into place.

Plain server installation -- updating the server

(information to be updated)

Currently deploying changes to production is analogous to merging into the rbx branch and pulling from the server.

Once this has been done, it is important that any database migrations have been applied. This can be verified by refreshing the server (calling a controller) and then looking at the database.

Step by Step

Resetting user passwords

Currently there is not easy way of resetting a user's password. The current method is the following:

cd server/web2py
python web2py -S vidjil -M db.auth_user[<user-id>].update_record(password=CRYPT(key=auth.settings.hmac_key)('<password>')[0],reset_password_key='')

Migrating Data

Usually, when extracting data for a given user or group, the whole database should not be copied over. The migrator script allows the selective export and import of data, whether it be a single patient/run/set or a list of them, or even all the sample sets associated to a group (or to a user). The script takes care both of database, but also of results and analysis files (see below for sequence files).

See server/web2py/applications/vidjil/scripts/migrator.py --help

Exporting an archive

(to be detailed)

Importing an archive

Step 1 : extract the archive on your server

The export directory must be on your server and accessible from your vidjil-server docker container. You can define a new shared volume; or simply put the export directory on an already accessible location such as [DOCKER DIRECTORY]/vidjil-server/conf/export/

Step 2 : prepare the group that will own the data

The permissions on a vidjil server are group based. Users and groups may be different from one server to another one. Before importing data on a server, one must have a group ready to receive the permissions to manage the imported files.

From the admin web interface has, you can create a new group ("groups" -> "+new group" -> "add group"). The group ID is displayed between parenthesis next to its name on the group page, you will need it later. If you create such a group on a blank vidjil server, the ID is 4.

Step 3 : prepare your server analysis configs

This step may require bioinformatics support depending on your data, the config previously used, and the ones you intend to use on your new installation. We can offer support via the VidjilNet consortium for help on setting that.

Vidjil analysis configs should not be directly transferred between servers. Indeed, they depend on the setup of each server setup (software, paths...) and can collide with existing configs on your installation. Before importing, you thus need to create the missing analysis configs on your server and edit the config.json file provided in the export folder.

This config.json file initially contains a list of the analysis configs from the original public server, such as:

  "2": {
      "description": [
        "IGH",
        "vidjil",
        "-c clones -3 -z 100 -r 1 -g germline/homo-sapiens.g:IGH,IGK,IGL,TRA,TRB,TRG,TRD -e 1 -w 50 -d -y all",
        "-t 100 -d lenSeqAverage",
        "multi-locus"
      ],
      "link_local": 6
  },

In the config.json file, you have to replace alllink_local values with the corresponding config ID of a similar config on your server (if you don't have a similar one you should create one).

If you much of your imported data was on old configs, that you do not intend to run anymore, a solution is to create a generic legacy config for these old data.

Below is an example of such a config.json, linking actual configuration on the public >app.vidjil.org> server to configs to a newly installed server. This should be completed by a mapping of other configs that were used in the migrated data.

{
  "2": {
    "description": [ "IGH", "vidjil",  "-c clones -3 -z 100 -r 1 -g germline/homo-sapiens.g:IGH,IGK,IGL,TRA,TRB,TRG,TRD -e 1 -w 50 -d -y all", "-t 100 -d lenSeqAverage",  "multi-locus" ],
    "link_local": 6
  },
  "25": {
    "description": [ "multi+inc+xxx",  "vidjil",  "-c clones -3 -z 100 -r 1 -g germline/homo-sapiens.g -e 1 -2 -d -w 50 -y all",  "-t 100 -d lenSeqAverage",  "default: multi-locus, with some incomplete/unusual/unexpected recombinations"
    ],
    "link_local": 2
  },
  "26": {
    "description": [ "multi+inc", "vidjil", "c clones -3 -z 100 -r 1 -g germline/homo-sapiens.g -e 1 -d -w 50",  "-t 100",  "multi-locus, with some incomplete/unusual recombinations" ],
    "link_local": 3
  },
  "30": {
    "description": [
      "TRG", "vidjil", "-c clones -3 -z 100 -r 1 -g germline/homo-sapiens.g:TRG -y all", "-t 100 -d lenSeqAverage", "TRG, VgJg"
    ],
    "link_local": 5
  },
  "32": {
    "description": [ "multi", "vidjil", "-c clones -3 -z 100 -r 1 -g germline/homo-sapiens.g:IGH,IGK,IGL,TRA,TRB,TRG,TRD -e 1 -w 50 -d -y all", "-t 100 -d lenSeqAverage", "multi-locus" ],
    "link_local": 4
  }
}

Step 4 : prepare your server pre-process configs

Proceed as in step 3 for pre-process configs. The file to edit is named pprocess.json.

Step 5 : import

The import takes place inside the vidjil-server container

docker exec -it docker_uwsgi_1 bash
cd usr/share/vidjil/server/web2py/applications/vidjil/scripts/
sh migrator.sh -p [RESULTS DIRECTORY] -s [EXPORT DIRECTORY] import --config [CONFIG.JSON FILE] --pre-process [PPROCESS.JSON FILE] [GROUP ID]

Usually, the command is thus:

sh migrator.sh -p /mnt/result/results/ -s /etc/vidjil/export/XXXX/ import --config/etc/vidjil/exportXXXX/config.json --pre-process /etc/vidjil/export/XXXX/pprocess.json  4

Exporting/importing input sequence files

Note that web2py and the Vidjil server are robust to missing input files. These files are not backuped and may be removed from the server at any time. Most of the time, these large files won't be migrated along with the database, the results and the analysis files.

However, they can simply be copied over to the new installation. Their filenames are stored in the database and should therefore be accessible as long as they are in the correct directories.

Exporting/importing a full database

When a full database migration is needed, it can be done with the following command:

mysqldump -u <user> -p <db> -c --no-create-info > <file>

The --no-create-info option is important because web2py needs to be allowed to create tables itself. Indeed, it keeps track of database migrations and errors will occur if tables exist which it considers it needs to create.

In order to import the data into an installation you first need to ensure the tables have been created by Web2py this can be achieved by simply accessing a non-static page.

/!\ If the database has been initialised from the interface you will likely encounter primary key collisions or duplicated data, so it is best to skip the initialisation altogether.

Once the tables have been created, the data can be imported as follows:

mysql -u <user> -p <db> < <file>

At least the results and analysis files should thus be copied.

Please note that with this method you should have at least one admin user that is accessible in the imported data. Since the initialization is being skipped, the usual admin account won't be present. It is also possible to create a user directly from the database although this is not the recommended course of action.

Using CloneDB [Under development]

The CloneDB has to be installed independently of the Vidjil platform.

Then one can easily extract data to be used with CloneDB. A script is provided (server/web2py/applications/vidjil/scripts/create_clone_db.py) which produces a FASTA file to be indexed with CloneDB. This script takes as parameter the FASTA output file and one (or many) group IDs, which correspond to the groups having access to the datasets. Note that for the moment the Vidjil platform only allow a per group access to the CloneDB.

The FASTA output filename must follow the format clonedb_XXX.fa where XXX is replaced with the group ID.

Make sure that the DIR_CLONEDB variable is set in defs.py and points to the CloneDB server directory. Make sure that in this directory the clonedb_defs.py has been filled correctly.

Then index the created FASTA file with the CloneDB index (follow the instructions from CloneDB).