Path: blob/main/documentation/content/en/books/developers-handbook/testing/_index.po
18099 views
# SOME DESCRIPTIVE TITLE # Copyright (C) YEAR The FreeBSD Project # This file is distributed under the same license as the FreeBSD Documentation package. # FIRST AUTHOR <EMAIL@ADDRESS>, YEAR. # #, fuzzy msgid "" msgstr "" "Project-Id-Version: FreeBSD Documentation VERSION\n" "POT-Creation-Date: 2025-05-01 19:56-0300\n" "PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n" "Last-Translator: FULL NAME <EMAIL@ADDRESS>\n" "Language-Team: LANGUAGE <[email protected]>\n" "Language: \n" "MIME-Version: 1.0\n" "Content-Type: text/plain; charset=UTF-8\n" "Content-Transfer-Encoding: 8bit\n" #. type: Title = #: documentation/content/en/books/developers-handbook/testing/_index.adoc:1 #: documentation/content/en/books/developers-handbook/testing/_index.adoc:15 #, no-wrap msgid "Regression and Performance Testing" msgstr "" #. type: YAML Front Matter: title #: documentation/content/en/books/developers-handbook/testing/_index.adoc:1 #, no-wrap msgid "Chapter 6. Regression and Performance Testing" msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:53 msgid "" "Regression tests are used to exercise a particular bit of the system to " "check that it works as expected, and to make sure that old bugs are not " "reintroduced." msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:55 msgid "" "The FreeBSD regression testing tools can be found in the FreeBSD source tree " "in the directory [.filename]#src/tools/regression#." msgstr "" #. type: Title == #: documentation/content/en/books/developers-handbook/testing/_index.adoc:57 #, no-wrap msgid "Micro Benchmark Checklist" msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:60 msgid "" "This section contains hints for doing proper micro-benchmarking on FreeBSD " "or of FreeBSD itself." msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:62 msgid "" "It is not possible to use all of the suggestions below every single time, " "but the more used, the better the benchmark's ability to test small " "differences will be." msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:64 msgid "Disable APM and any other kind of clock fiddling (ACPI ?)." msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:65 msgid "" "Run in single user mode. E.g., man:cron[8], and other daemons only add " "noise. The man:sshd[8] daemon can also cause problems. If ssh access is " "required during testing either disable the SSHv1 key regeneration, or kill " "the parent `sshd` daemon during the tests." msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:66 msgid "Do not run man:ntpd[8]." msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:67 msgid "" "If man:syslog[3] events are generated, run man:syslogd[8] with an empty " "[.filename]#/etc/syslogd.conf#, otherwise, do not run it." msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:68 msgid "Minimize disk-I/O, avoid it entirely if possible." msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:69 msgid "Do not mount file systems that are not needed." msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:70 msgid "" "Mount [.filename]#/#, [.filename]#/usr#, and any other file system as read-" "only if possible. This removes atime updates to disk (etc.) from the I/O " "picture." msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:71 msgid "" "Reinitialize the read/write test file system with man:newfs[8] and populate " "it from a man:tar[1] or man:dump[8] file before every run. Unmount and mount " "it before starting the test. This results in a consistent file system " "layout. For a worldstone test this would apply to [.filename]#/usr/obj# " "(just reinitialize with `newfs` and mount). To get 100% reproducibility, " "populate the file system from a man:dd[1] file (i.e.: `dd if=myimage of=/dev/" "ad0s1h bs=1m`)" msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:72 msgid "Use malloc backed or preloaded man:md[4] partitions." msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:73 msgid "" "Reboot between individual iterations of the test, this gives a more " "consistent state." msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:74 msgid "" "Remove all non-essential device drivers from the kernel. For instance if USB " "is not needed for the test, do not put USB in the kernel. Drivers which " "attach often have timeouts ticking away." msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:75 msgid "" "Unconfigure hardware that are not in use. Detach disks with " "man:atacontrol[8] and man:camcontrol[8] if the disks are not used for the " "test." msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:76 msgid "" "Do not configure the network unless it is being tested, or wait until after " "the test has been performed to ship the results off to another computer." msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:78 msgid "" "Disable \"Turbo-modes\" because they make the clock frequency explicitly " "depend on the environment. This means that benchmark runs on 100% identical " "code, may depend on time of day, coffee vs. soda or even how many other " "people are in the office." msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:80 msgid "" "If the system must be connected to a public network, watch out for spikes of " "broadcast traffic. Even though it is hardly noticeable, it will take up CPU " "cycles. Multicast has similar caveats." msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:81 msgid "" "Put each file system on its own disk. This minimizes jitter from head-seek " "optimizations." msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:82 msgid "" "Minimize output to serial or VGA consoles. Running output into files gives " "less jitter. (Serial consoles easily become a bottleneck.) Do not touch " "keyboard while the test is running, even kbd:[space] or kbd:[back-space] " "shows up in the numbers." msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:83 msgid "" "Make sure the test is long enough, but not too long. If the test is too " "short, timestamping is a problem. If it is too long temperature changes and " "drift will affect the frequency of the quartz crystals in the computer. Rule " "of thumb: more than a minute, less than an hour." msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:84 msgid "" "Try to keep the temperature as stable as possible around the machine. This " "affects both quartz crystals and disk drive algorithms. To get real stable " "clock, consider stabilized clock injection. E.g., get a OCXO + PLL, inject " "output into clock circuits instead of motherboard xtal. Contact {phk} for " "more information about this." msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:85 msgid "" "Run the test at least 3 times but it is better to run more than 20 times " "both for \"before\" and \"after\" code. Try to interleave if possible (i.e.: " "do not run 20 times before then 20 times after), this makes it possible to " "spot environmental effects. Do not interleave 1:1, but 3:3, this makes it " "possible to spot interaction effects." msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:88 msgid "" "A good pattern is: `bababa{bbbaaa}*`. This gives hint after the first 1+1 " "runs (so it is possible to stop the test if it goes entirely the wrong way), " "a standard deviation after the first 3+3 (gives a good indication if it is " "going to be worth a long run) and trending and interaction numbers later on." msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:89 msgid "" "Use man:ministat[1] to see if the numbers are significant. Consider buying " "\"Cartoon guide to statistics\" ISBN: 0062731025, highly recommended, if you " "have forgotten or never learned about standard deviation and Student's T." msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:90 msgid "" "Do not use background man:fsck[8] unless the test is a benchmark of " "background `fsck`. Also, disable `background_fsck` in [.filename]#/etc/" "rc.conf# unless the benchmark is not started at least 60+\"``fsck`` " "runtime\" seconds after the boot, as man:rc[8] wakes up and checks if `fsck` " "needs to run on any file systems when background `fsck` is enabled. " "Likewise, make sure there are no snapshots lying around unless the benchmark " "is a test with snapshots." msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:91 msgid "" "If the benchmark show unexpected bad performance, check for things like high " "interrupt volume from an unexpected source. Some versions of ACPI have been " "reported to \"misbehave\" and generate excess interrupts. To help diagnose " "odd test results, take a few snapshots of `vmstat -i` and look for anything " "unusual." msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:92 msgid "" "Make sure to be careful about optimization parameters for kernel and " "userspace, likewise debugging. It is easy to let something slip through and " "realize later the test was not comparing the same thing." msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:93 msgid "" "Do not ever benchmark with the `WITNESS` and `INVARIANTS` kernel options " "enabled unless the test is interested to benchmarking those features. " "`WITNESS` can cause 400%+ drops in performance. Likewise, userspace " "man:malloc[3] parameters default differently in -CURRENT from the way they " "ship in production releases." msgstr "" #. type: Title == #: documentation/content/en/books/developers-handbook/testing/_index.adoc:95 #, no-wrap msgid "The FreeBSD Source Tinderbox" msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:98 msgid "The source Tinderbox consists of:" msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:100 msgid "" "A build script, [.filename]#tinderbox#, that automates checking out a " "specific version of the FreeBSD source tree and building it." msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:101 msgid "" "A supervisor script, [.filename]#tbmaster#, that monitors individual " "Tinderbox instances, logs their output, and emails failure notices." msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:102 msgid "" "A CGI script named [.filename]#index.cgi# that reads a set of tbmaster logs " "and presents an easy-to-read HTML summary of them." msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:103 msgid "" "A set of build servers that continually test the tip of the most important " "FreeBSD code branches." msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:104 msgid "" "A webserver that keeps a complete set of Tinderbox logs and displays an up-" "to-date summary." msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:107 msgid "" "The scripts are maintained and were developed by {des}, and are now written " "in Perl, a move on from their original incarnation as shell scripts. All " "scripts and configuration files are kept in https://www.freebsd.org/cgi/" "cvsweb.cgi/projects/tinderbox/[/projects/tinderbox/]." msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:109 msgid "" "For more information about the tinderbox and tbmaster scripts at this stage, " "see their respective man pages: tinderbox(1) and tbmaster(1)." msgstr "" #. type: Title == #: documentation/content/en/books/developers-handbook/testing/_index.adoc:110 #, no-wrap msgid "The index.cgi Script" msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:117 msgid "" "The [.filename]#index.cgi# script generates the HTML summary of tinderbox " "and tbmaster logs. Although originally intended to be used as a CGI script, " "as indicated by its name, this script can also be run from the command line " "or from a man:cron[8] job, in which case it will look for logs in the " "directory where the script is located. It will automatically detect " "context, generating HTTP headers when it is run as a CGI script. It " "conforms to XHTML standards and is styled using CSS." msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:120 msgid "" "The script starts in the `main()` block by attempting to verify that it is " "running on the official Tinderbox website. If it is not, a page indicating " "it is not an official website is produced, and a URL to the official site is " "provided." msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:123 msgid "" "Next, it scans the log directory to get an inventory of configurations, " "branches and architectures for which log files exist, to avoid hard-coding a " "list into the script and potentially ending up with blank rows or columns. " "This information is derived from the names of the log files matching the " "following pattern:" msgstr "" #. type: delimited block . 4 #: documentation/content/en/books/developers-handbook/testing/_index.adoc:127 #, no-wrap msgid "tinderbox-$config-$branch-$arch-$machine.{brief,full}\n" msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:131 msgid "" "The configurations used on the official Tinderbox build servers are named " "for the branches they build. For example, the `releng_8` configuration is " "used to build `RELENG_8` as well as all still-supported release branches." msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:133 msgid "" "Once all of this startup procedure has been successfully completed, " "`do_config()` is called for each configuration." msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:135 msgid "" "The `do_config()` function generates HTML for a single Tinderbox " "configuration." msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:138 msgid "" "It works by first generating a header row, then iterating over each branch " "build with the specified configuration, producing a single row of results " "for each in the following manner:" msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:140 msgid "For each item:" msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:142 msgid "For each machine within that architecture:" msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:144 msgid "If a brief log file exists, then:" msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:146 msgid "Call `success()` to determine the outcome of the build." msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:147 msgid "Output the modification size." msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:148 msgid "" "Output the size of the brief log file with a link to the log file itself." msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:149 msgid "If a full log file also exists, then:" msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:151 msgid "" "Output the size of the full log file with a link to the log file itself." msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:153 msgid "Otherwise:" msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:155 msgid "No output." msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:157 msgid "" "The `success()` function mentioned above scans a brief log file for the " "string \"tinderbox run completed\" in order to determine whether the build " "was successful." msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:160 msgid "" "Configurations and branches are sorted according to their branch rank. This " "is computed as follows:" msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:162 msgid "`HEAD` and `CURRENT` have rank 9999." msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:163 msgid "`RELENG_x` has rank __``xx``__99." msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:164 msgid "`RELENG_x_y` has rank _xxyy_." msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:167 msgid "" "This means that `HEAD` always ranks highest, and `RELENG` branches are " "ranked in numerical order, with each `STABLE` branch ranking higher than the " "release branches forked off of it. For instance, for FreeBSD 8, the order " "from highest to lowest would be:" msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:169 msgid "`RELENG_8` (branch rank 899)." msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:170 msgid "`RELENG_8_3` (branch rank 803)." msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:171 msgid "`RELENG_8_2` (branch rank 802)." msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:172 msgid "`RELENG_8_1` (branch rank 801)." msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:173 msgid "`RELENG_8_0` (branch rank 800)." msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:177 msgid "" "The colors that Tinderbox uses for each cell in the table are defined by " "CSS. Successful builds are displayed with green text; unsuccessful builds " "are displayed with red text. The color fades as time passes since the " "corresponding build, with every half an hour bringing the color closer to " "grey." msgstr "" #. type: Title == #: documentation/content/en/books/developers-handbook/testing/_index.adoc:178 #, no-wrap msgid "Official Build Servers" msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:181 msgid "" "The official Tinderbox build servers are hosted by http://" "www.sentex.ca[Sentex Data Communications], who also host the FreeBSD Netperf " "Cluster." msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:183 msgid "Three build servers currently exist:" msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:185 msgid "_freebsd-current.sentex.ca_ builds:" msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:187 msgid "" "`HEAD` for amd64, arm, i386, i386/pc98, ia64, mips, powerpc, powerpc64, and " "sparc64." msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:188 msgid "" "`RELENG_9` and supported 9._X_ branches for amd64, arm, i386, i386/pc98, " "ia64, mips, powerpc, powerpc64, and sparc64." msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:190 msgid "_freebsd-stable.sentex.ca_ builds:" msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:192 msgid "" "`RELENG_8` and supported 8._X_ branches for amd64, i386, i386/pc98, ia64, " "mips, powerpc and sparc64." msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:194 msgid "_freebsd-legacy.sentex.ca_ builds:" msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:196 msgid "" "`RELENG_7` and supported 7._X_ branches for amd64, i386, i386/pc98, ia64, " "powerpc, and sparc64." msgstr "" #. type: Title == #: documentation/content/en/books/developers-handbook/testing/_index.adoc:197 #, no-wrap msgid "Official Summary Site" msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:200 msgid "" "Summaries and logs from the official build servers are available online at " "http://tinderbox.FreeBSD.org[http://tinderbox.FreeBSD.org], hosted by {des} " "and set up as follows:" msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:202 msgid "" "A man:cron[8] job checks the build servers at regular intervals and " "downloads any new log files using man:rsync[1]." msgstr "" #. type: Plain text #: documentation/content/en/books/developers-handbook/testing/_index.adoc:202 msgid "Apache is set up to use [.filename]#index.cgi# as `DirectoryIndex`." msgstr ""