Note
Here are aggregated notes forming a part of the developer documentation on the Vidjil web client.
These notes are a work-in-progress, they are not as polished as the user documentation.
Developers should also have a look at the documentation for bioinformaticians and server administrators, at the issues, at the commit messages, and at the source code.
Development notes -- ClientβοΈ
InstallationβοΈ
Run a make
into browser/
to get the necessary files.
This will in particular get the germline files as well as the icon files.
Opening the browser/index.html
file is then enough to get a functionning client,
able to open .vidjil
files with the import/export
dev-ger menu.
To work with actual data, the easiest way is to copy js/conf.js.sample
to js/conf.js
.
This will unlock the patients
menu and allow your local client
to access the public server at http://app.vidjil.org/.
Installation with DockerβοΈ
This section is intended for people wanting to install a vidjil client using docker WITHOUT a vidjil server. If you wish to install a vidjil server altogether with your client please refer to the docker section in dev-server.md.
SetupβοΈ
The config file that will be used by the client can be found at vidjil-client/conf/conf.js
Since this installation does not provide a Vidjil server it is recommended to disable the use of databases,
"use_database" : false,
or to provide an URL to connect to an existing one online.
"db_address" : "https://VIDJILSERVERURL/vidjil",
Starting the environmentβοΈ
The vidjil Docker environment is managed by Docker Compose since it is composed of several different services, but a single service, nginx, is required to run the vidjil client, for a more detailed explanation on other services see dev-server.md.
Ensure your docker-compose.yml contains the correct reference to the vidjil-client image you want to use. Usually this will be vidjil/vidjil-client:latest, but more tags are available at https://hub.docker.com/r/vidjil/vidjil/tags/.
Running the following command will automatically download any missing images and start the environment:
docker-compose up nginx
Client API and permanent URLsβοΈ
The client can be opened on a data file specified from a data
attribute,
and optionally on an analysis file specified from a analysis
attribute,
as in the following URLs on our test server:
- http://app.vidjil.org/browser/?data=test.vidjil
- http://app.vidjil.org/browser/?data=test.vidjil&analysis=test.analysis
- http://app.vidjil.org/browser/?data=http://app.vidjil.org/browser/test.vidjil
Both GET and POST requests are accepted.
Note that the browser/index.html
file and the .vidjil/.analysis
files should be hosted on the same server.
Otherwise, the server hosting the .vidjil/.analysis
files must accept cross-domain queries.
The client can also load data from a server (see below, requires logging) using url parameters to pass file identifiers, as in http://app.vidjil.org/?set=3241&config=39
set=xx |
sample set id |
config=yy |
config id |
or directly inside the URL for a shortened version, as in http://app.vidjil.org/3241-39/
Older formats (patients, runβ¦) are also supported for compatibility but deprecated. Moreover, the state of the client can be encoded in the URL, as in http://app.vidjil.org/3241-39/?plot=v,size,bar&clone=11,31
plot=x,y,m |
plot (x axis, y axis) |
clone=xx,xx,xx |
selected clone ids |
For plot
the axis names are found in browser/js/axis_conf.js
. m
is optional, and defines the type of plot (either grid
or bar
).
We intend to encode more parameters in the URL.
ArchitectureβοΈ
The Vidjil client is a set of views linked to a same model. The model keeps the views in sync on some global properties, most notably dealing with the selection of clones, with the clone filtering, as well with the locus selection.
-
The model (
js/model.js
) is the main object of the Vidjil client. It loads and saves.vidjil
json data (either directly from data, or from a local file, or from some url). It provides function to access and edit information on the clones and on the global parameters It keeps all the views in sync. -
Each of the views (
Graph
,ScatterPlot
,List
,Segment
) is rendered inside one or several<div>
elements, and kept sync with the model. All the views are optional, and several views of the same type can be added. Seejs/main.js
for the invocation -
The link with the patient database/server is done with the
Database
object (js/database.js
) -
Other objects:
Report
,Shortcut
Extends functionalities but requires elements from the fullindex.html
.
Clone attributesβοΈ
Clone use boolean mask in order to specify attributes.
These attributes allow specific behavior inside the client: C_SIZE_CONSTANT
, C_SIZE_DISTRIB
and C_SIZE_OTHER
.
Each clone has one (and only one) attributes linked to his type/size.
Either its raw size is constant (_CONSTANT
), either it is computed from other clones (_DISTRIB
, _OTHER
).
Note that, that in each case, the displayed size can be different from the raw size
due to normalizations.
Attributes | Effect |
---|---|
C_INTERACTABLE | The clone can be selected by the user. It can be focused in, or hidden. |
C_CLUTERIZABLE | The clone can be clustered with other clones. This clone should also be C_INTERACTABLE. |
C_INSCATTERPLOT | Each clone that has values on the current axes will be displayed in the 'scatterplot' panel. |
.sequence | Each clone that has a sequence will be displayed in the bottom 'segmenter' panel. |
Creation of common clonesβοΈ
Three types of clone are now created combining some of the attributes
In the following example, data
is an map specifying values for a clone (locus, segments, number of reads, ...).
-
Constant
clone: actual individual clones, described into an item of theclones
list in the.vidjil
file// Constant new Clone(data, model, index, C_SIZE_CONSTANT | C_CLUSTERIZABLE | C_INTERACTABLE | C_IN_SCATTERPLOT);
-
Other
, or smaller clones: corresponding to the sum of each clones of a given locus, with size dynamically computed to take into account the current filter and viewable constant clones.// Other new Clone(data, model, index, C_SIZE_OTHER);
-
Distribution
clones correspond todistributions
lists of the.vidjil
file. See "distributions" invidjil-format.md
. They are aggregate information on clones that won't be shown individually in the client, and are useful to display views such a "simulated Genescan". They are generated bymodel.loadAllDistribClones()
, that agregates such data for each sample.// Distributions new Clone(data, model, index, C_SIZE_DISTRIB | C_INTERACTABLE | C_IN_SCATTERPLOT );
Update mechanismβοΈ
Views object are used to display the content of an Object.model() The views are using 3 different update functions to synchronyze the data stored in the model and the content they display. If you make any change to the model you should use one of these functions to resync the view with it.
updateElemStyle(list[])βοΈ
What does it do:
UpdateElemStyle take a list of clone ID and update the look of those clones in the view.
really fast, this usually does not need any complex redraw and is dealt with a simple change in a css attribute.
When should it be used:
- if the clone state(selected/onFocus/hidden/...) has changed but not his data values (name/size/v/d/j/...)
- if the colorMethod used in the model has changed
updateElem(list[])βοΈ
What does it do:
UpdateElem take a list of clone ID and update them fully in the view.
When should it be used:
- after an operation that modify a clone data values(name/size/v/d/j/...)
- after a clone rename or any clustering operations for example.
update()βοΈ
What does it do:
redraw the view from scratch.
When should it be used:
- after a change in the view parameters (like the selected axis or the scale used)
- if a change on how is computed the clones data values(name/size/v/d/j/...) is made. (for example, anything that change the results of the getSize() or getName() functions for the clones, like a change in the germline selection, the normalization or the selected sample)
model updatesβοΈ
Most of the time you will not call the view updates functions but the model equivalent that take care to call the updates on all the views linked to it.
m.update()
m.updateElem()
m.updateElemList()
Integrating the clientβοΈ
HTML and CSSβοΈ
- The
index.html
contains the<div>
for all views and the menus - The CSS (
css/light.css
) is generated byless
fromcss/vidjil.less
- The
small_example.html
is a minimal example embedding basic HTML, CSS, as well as some data. As the menus are not embedded in this file, functionalities should be provided by direct calls to the models and the views.
JavascriptβοΈ
- The wonderful library
require.js
is used, so there is only one file to include \<script data-main="js/app.js" src="js/lib/require.js">\</script> js/main.js
creates the different views and binds them to the model. Another option is to directly define a function namedmain()
, as insmall_example.html
.
JSON .vidjil dataβοΈ
Clone lists can be passed to the model through several ways:
- directly by the user (import/export)
- from a patient database (needs a database)
- trough the API (see below)
- or by directly providing data through Javascript (as in
small_example.html
)
The first three solutions need some further elements from the full index.html
.
NotificationsβοΈ
PriorityβοΈ
# The priority determines how the notification are shown and what action the user should do. The priorities can be between 0 and 3.
Level | Effects |
---|---|
0 | The notification is not shown |
1 | The notification is shown (usually on green background) and automatically disappears |
2 | The notification is shown (usually on yellow background) and automatically disappears |
3 | The notification is shown (usually on red background) and doesn't disappear until the user clicks on it |
In the console.log
, the field priority
takes one of those priorities.
PlotsβοΈ
How to add something to be plottedβοΈ
You want to add a dimension in the scatterplot or as a color? Read the following.
-
Axis
In axes.js, the
AXIS_DEFAULT
object defines the dimensions that can be displayed. It suffices to add an entry so that it will be proposed in the X and Y axis. This kind of way of doing should be generalized to the other components.Here is some of the settings you can use to customize your axis. - name a short description of the axis - doc a more detailled description of the axis - fct a javascript function that must return a value to be displayed on the axis for a given clone ID - scale used to define numerical axis min/max value - labels a list of labels that must always be present on the axis even if no clones has returned the corresponding value. - autofill autofill : true mean the list of label will be created or extended with all unique values returned by the clones. It will also create an adapted scale with the min/max value returned by the clones in case of a numerical axis. - sort you can provide a custom comparison function to sort the labels in a specific order
There is also other settings that can be used to customize even further labels appearance or display, please check the already defined axes in [axes.js] to learn more about them.
-
Preset
The presets are defined in the
preset
object that can be found in [scatterPlot_menu.js]. -
Color
Adding a color needs slightly more work than adding a dimension in the scatterplot.
The function
updateColor
in file clone.js must be modified to add our color method. The variablethis.color
must contain a color (either in HTML or RGB, orβ¦).Then a legend must be displayed to understand what the color represents. For this sake, modify the
build_info_color
method in info.js file. By default four spans are defined (that can be used) to display the legend:span0
, β¦,span3
.Finally modify the index.html file to add the new color method in the select box (which is under the
color_menu
ID).
Sequence panelβοΈ
Add a sequence featureβοΈ
A sequence feature can be used to highlight a specific part of a sequence. Here for example is the sequence feature describing how to highlight the V region as available in aligner_layer.js
'V': {
'title': function (s,c) { return c.seg["5"].name;},
'start': function (s,c) { return c.getSegStart("5"); },
'stop': function (s,c) { return c.getSegStop("5"); },
'className': "seq_layer_highlight",
'style': { 'background': "#4c4" },
'enabled': true
}
- title : [text] the content of the html title field of the feature.
- start : [int] the position of the first nucleotide of the selected region.
- stop : [int] the position of the last nucleotide of the selected region .
- text : [int] (optional) text to overlay on top of the sequence.
- condition : [boolean] (optional) sequence feature will be displayed only if true.
- classname : [text] (optional) html classname used to customize the sequence feature look.
- style : [object] (optional) additional css properties to further customize the sequence feature.
- enabled : [boolean] default visibility
most field can take a static value or a function that will be able to return a specific value for each clone.
function (s,c) { ...}
- s : the aligner_sequence object (check aligner_sequence.js to see available functions)
- c : the clone object (check clone.js to see functions / data available)
How to add a sequence feature in the menuβοΈ
You can set the 'enabled' sequence feature field to true to always display it, or, you can edit the aligner_menu file to add an entry to the sequence panel menu allowing you to enable/disable your sequence feature with a checkbox.
example : the aligner_menu.js entry allowing to enable/disable the V/D/J regions of the sequence
{
'text': 'V/D/J genes',
'title': 'Highlight V/D/J genes',
'layers': ["V","D","J"],
'enabled': true
}
- text : [text] checkbox text to display in the sequence menu panel
- title : [text] the content of the html title field of the checkbox.
- layers : [array] a list of sequence feature name defined in aligner_layer.js to enable/disable
- enabled : [boolean] default checkbox value
ClassesβοΈ
CloneβοΈ
Info box
In the info box all the fields starting with a \_ are put. Also all the
fields under the `seg` field are displayed as soon as they have a `start` and
`stop`. Some of them can be explicitly not displayed by filling the
`exclude_seg_info` array in `getHtmlInfo`.
TestsβοΈ
Code QualityβοΈ
Quality of code is checked using JSHint, by
running make quality
from the browser
directory.
Install with npm install -g jshint
UnitβοΈ
The unit tests in the client are managed by QUnit and launched using
nightmare, by launching make unit
from the browser/test
directory.
The tests are organised in the directory
browser/test/QUnit/testFiles. The file datatest.js contains a toy
dataset that is used in the tests.
Unit tests can be launched using a real client (instead of nightmare). It suffices to open the file testQunit.html. In this HTML webpage it is possible to see the coverage. It is important that all possible functions are covered by unit tests. Having the coverage displayed under Firefox needs to display the webpage using a web server for security reasons. Under Chromium/Chrome this should work fine by just opening the webpage.
InstallationβοΈ
Nightmare is distributed withing node
and can therefore be installed with it.
apt-get install nodejs-legacy npm
npm install nightmare -g # make -C browser/test unit will automatically
link to global nightmare installation
Note that using nightmare
for our unit testing
requires the installation of xvfb
.
DebuggingβοΈ
If there is a problem with the nightmare or electron (nightmare dependency), you may encounter a lack of output or error messages. To address this issue, run:
cd browser/test/QUnit
DEBUG=nightmare*,electron:* node nightmare.js
Functional tests with WatirβοΈ
Warning
Watir tests is currently under suppression and it is replaced by Cypress testing pipeline.
Migration of client side is already done, and server side is in progress.
Please go to Functional tests with Cypress for server/client testing.
Some part of documentation on Watir still true, notably for installation and server testing part, but a major part of it is no longer usable.
The documentation remains until Watir is completely removed.
ArchitectureβοΈ
The client functional testing is done in the directory
browser/tests/functional
, with Watir.
The functional tests are built using two base files:
vidjil_browser.rb
abstracts the vidjil browser (avoid using IDs or class names that could change in the test). The tests must rely as much as possible onvidjil_browser
. If access to some data/input/menus are missing they must be addded there.browser_test.rb
prepares the environment for the tests. Each test file will extend this class (as can be seen intest_multilocus.rb
)
The file segmenter_test.rb
extends the class in browser_test.rb
to adapt
it to the purpose of testing the analyze autonomous app.
The tests are in the files whose name matches the pattern test*.rb
. The
tests are launched by the script in ../launch_functional_tests
which launches
all the files matching the previous pattern. It also backs up the test
reports as ci_reporter
removes them before each file is run.
InstallationβοΈ
The following instructions are for Ubuntu. For OS X, see https://github.com/watir/watirbook/blob/master/manuscript/installation/mac.md.
Install rvm
curl -sSL https://get.rvm.io | sudo bash
Afterwards you may need to launch:
source /etc/profile.d/rvm.sh
Install ruby 2.6.1
rvm install 2.6.1
Switch to ruby 2.6.1
rvm use 2.6.1
Install necessary gems
gem install minitest minitest-ci watir test-unit
Install web browsers
The Firefox version used can be set with an environment variable (see below). By default, the tests only work with Firefox β€ 45. But this can be modified with an environment variable. All Firefox releases are available here.
One can instead choose to launch functional tests using chrome. You should
install chromium-browser
as well as chromium-chromedriver
. On old Chrome
versions the chromedriver
package may not exist. In such a case you should
download the ChromeDriver by yourself (the supported Chrome versions are
written in the notes.txt
file of each version).
Launch client testsβοΈ
As indicated previously, rvm
must have been loaded. Thus you may need to
first launch:
source /etc/profile.d/rvm.sh
rvm use 2.6.1
Then you can launch the tests (potentially by altering its behaviour with environment variables, see below).
make functional
Environment variables
By default the tests are launched on the Firefox installed on the system.
This can be modified by providing the WATIR_BROWSER_PATH
environment
variable that defines the path to another Firefox version.
Other environment variables that can be specified to tune the default behaviour
WATIR_CHROME
should be set to something evaluated toTrue
(e.g. 1) if the tests must be launched using Chrome (in such a caseWATIR_BROWSER_PATH
is useless)WATIR_MARIONETTE
should be set to something evalued toTrue
(e.g. 1) if the tests must be launched on a recent Firefox version (> 45)
If you have set a configuration file (browser/js/conf.js), you should remove it during the tests. The easiest way to do it is to launch these commands before and after the tests
# before tests
mv browser/js/conf.js browser/js/conf.js.bak
# after tests
mv browser/js/conf.js.bak browser/js/conf.js
Headless modeβοΈ
On servers without a X server the client tests can be launched in headless mode. For this sake one needs to install a few more dependencies:
gem install headless
The virtual framebuffer X server (xvfb
) must also be installed. Depending
on the operating system the command will be different:
# On Debian/Ubuntu
apt-get install xvfb
# On Fedora/CentOS
yum install xvfb
Then the client tests can be launched in headless mode with:
make headless
It is possible to view the framebuffer content of Xvfb
using vnc
. To do so,
launch:
x11vnc -display :99 -localhost
vncviewer :0
This will work when the headless mode is launched on the local machine.
Otherwise, when one wants to view what is going on on a distant host, one
should additionally do port forwarding through SSH, with ssh -L
5900:localhost:5900 distant_host
, where 5900 is the port number on which
the VNC server is launched. Then vncviewer
can be launched locally with
vncviewer :5900
.
Interactive modeβοΈ
For debugging purposes, it may be useful to launch Watir in interactive
mode. In that case, you should launch irb
in the browser/tests/functional
directory.
Then load the file browser_test.rb
and create a BrowserTest
:
load 'browser_test.rb'
bt = BrowserTest.new "toto"
# Load the Vidjil client with the given .vidjil file
bt.set_browser("/doc/analysis-example.vidjil")
Finally you can directly interact with the VidjilBrowser
using the $b
variable.
Another way of debugging interactively is by using (and installing) the
ripl
gem. Then you should add, in the .rb
file to debug:
require 'ripl'
Then if you want to stop launch an irb
arrived at a given point in the
code, the following command must be inserted in the code:
Ripl.start :binding => binding
If you have to launch irb
on a remote server without X (only using Xvfb
)
you may be interested to use the redirection over SSH.
Functional tests with cypressβοΈ
The Cypress testing pipeline is build on a Docker image which include the following Chrome and Firefox browsers:
Firefox | Chromium | |
---|---|---|
Legacy (until september 2021) | 62.0 | 75.0.3770.0 |
Supported | 78.0 | 79.0.3945.0 |
Latest (as at june 2021) | 89.0 | 93.0.4524.0 |
We will progressivly convert historic Watir tests toward Cypress.
InstallationβοΈ
Install Docker, then either build locally the Docker image, or download it from dockerhub
Local build
docker build ./docker/ci -t "vidjilci/cypress_with_browsers:latest"
Dockerhub pull
docker pull "vidjilci/cypress_with_browsers:latest"
UsageβοΈ
By default, the cypress pipeline is launched in headless mode.
The makefile rule make functional_browser_cypress
launches the following command:
docker run \
--user $(id -u):$(id -g) \
-v `pwd`/browser/test/cypress:/app/cypress \
-v `pwd`/browser/test/data/:/app/cypress/fixtures/data/ \
-v `pwd`/doc/:/app/cypress/fixtures/doc/ \
-v `pwd`/demo/:/app/cypress/fixtures/demo/ \
-v `pwd`:/app/vidjil \
-v "`pwd`/docker/ci/cypress_script.bash":"/app/script.bash" \
-v "`pwd`/docker/ci/script_preprocess.bash":"/app/script_preprocess.bash" \
-v "`pwd`/docker/ci/cypress.json":"/app/cypress.json" \
--env BROWSER=electron --env HOST=localhost "vidjilci/cypress_with_browsers:latest" bash script.bash
Local volumes are mounted for these tests.
Tests scripts are located in browser/test/cypress
:
support
(shared functions)fixtures
(data used during tests)integration
(testing scripts)
Interactive modeβοΈ
The interactive mode allows to select tests to be launched. Cypress has to be installed on local computer. The following command creates some links and open the GUI:
make functional_browser_cypress_open
A test_sandbox
is available to quickly test some modification made in the browser. See file test_sandbox.js
and other script file for fast developpment.
TroubleshootingβοΈ
- Xvfb error
-
The cypress pipeline may fail in some cases, when, after the end of the tests, the Xvfb server and the docker container are still running. In this case, stop the docker container.
docker ps
docker stop $container_id
- Permission errors on report and screenshot files
-
Files produced by cypress docker belong to the root user. These files should be deleted with root privilege.
sudo rm -r browser/test/cypress/report browser/test/cypress/screenshots