Re: Continual benchmarking / regression testing?

Den 08/01/2014 kl. 17.38 skrev Alan Somers <asomers@FreeBSD.org>:
> =

> I like that you stored test results in a SQL database.  My SQL-foo is
> poor, so ATM my framework is using CSV.  It also looks like you've got
> code to generate a website.  Do you have any example output?

Yes, there=92s a website to filter results, generate graphs, see commit mes=
sages between two data points and show the hardware and software configurat=
ion of the client running the benchmark. A continuous benchmarking framewor=
k is only useful if it can assist you in analyzing the data, finding regres=
sions and their cause.

> The PXE stuff, however, does not belong in the
> benchmark framework, IMHO.  I think that the benchmark framework
> should just include the benchmarking and system profiling aspect, not
> system configuration.  Provisioning and configuring systems can be
> done in a separate utility, one that can be shared, for example, with
> the continuous Kyua tester.

System configuration affects benchmark results, so that needs to be recorde=
d along with the benchmark results. My work was intended as a complete cont=
inuous benchmarking system with a build machine that produces OS installati=
on images and tells clients what to install and what to run. But I agree th=
at a benchmark framework contains many self-contained parts that could be s=
hared among projects.

Erik
_______________________________________________
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"