Results can be selected and compared using the "Compare
Results" button (tab) (select a result by either clicking on
a row below, or result bullet above).
Nuances
Some things, annoyingly, don't quite work right:
-
new results slow to appear
The local browser
cache seems to get in the way. One solution is to
fetch the json files with ?<current-git-rev>
appended.
-
comparisons sometimes loose a result
The code
fetching all the json result tables is likely racy,
when too may results are clicked too fast a result
goes missing.
The work around is to de-select
and then re-select the missing result.
-
libreswan's repo contains "knots"
When
following a linear history (parent links), the commit
dates normally decrease. In this repo they some times
increase resulting in graph doubling back on itself.
-
libreswan's repo contains "plats"
As a
generalization, is good to merge local changes onto
the remote first-parent branch and not the reverse.
Unfortunately "git pull (git merge)" does the reverse
by default. The resut is that first-parent keeps
switching with development branches.
-
clicking links toggls a results selection
For
instance, control-click a result's subject hash link
(to open a new tab displaying the diff)) also toggls
the results appearance under the comparison tab
Maintenance
From time to time the test results are seen to decay - an
increasing number of tests fail with a result of
"unresolved" and an error of "output-missing". The
problem is domains taking too long to boot (over 2 minutes
each).
The fix is to restart the machine and then re-build the
test domains. It is also a good opportunity to perform
some more general maintenance.
-
upgrade and reboot the test machine:
dnf upgrade -y ; reboot
-
delete the existing domains:
cd $BUILDDIR && make kvm-uninstall
-
update the build tree to latest:
cd $BUILDDIR && git pull --ff-only
-
delete result directories with missing output:
cd $WEBDIR && grep '"output-missing"' */results.json | cut -d/ -f1 | sort -u # | xargs rm -rf
-
delete result directories with incomplete output:
cd $WEBDIR && ls -d */ | while read d ; do test -r $d/results.json || echo $d ; done # | xargs rm -rf
-
compress any log files:
cd $WEBDIR && find * -name '*.log' -print | xargs bzip2 -9
-
restart tester.sh:
cd && nohup $SCRIPTDIR/testing/web/tester.sh $BUILDDIR $WEBDIR
Some things could work better:
-
can't select individual tests for comparison
selecting an individual or small set of tests and
comparing them across results would be nice
-
can't bookmark a comparison
-
an accumulative bar graph is probably a better way to
represent the data (or at least the first-parents)
that could then be overlaid with a scatter plot
-
the graph should pan and zoom
-
better colour scheme welcome