<?xml version="1.0" encoding="UTF-8" standalone="no"?><!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"><html xmlns="http://www.w3.org/1999/xhtml"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /><title>Test</title><meta name="generator" content="DocBook XSL Stylesheets V1.73.2" /><meta name="keywords" content=" ISO C++ , test , testsuite " /><meta name="keywords" content=" ISO C++ , library " /><link rel="start" href="../spine.html" title="The GNU C++ Library Documentation" /><link rel="up" href="bk01pt01ch02.html" title="Chapter 2. Setup" /><link rel="prev" href="build.html" title="Build" /><link rel="next" href="using.html" title="Chapter 3. Using" /></head><body><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">Test</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="build.html">Prev</a> </td><th width="60%" align="center">Chapter 2. Setup</th><td width="20%" align="right"> <a accesskey="n" href="using.html">Next</a></td></tr></table><hr /></div><div class="sect1" lang="en" xml:lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a id="manual.intro.setup.test"></a>Test</h2></div></div></div><div class="sect2" lang="en" xml:lang="en"><div class="titlepage"><div><div><h3 class="title"><a id="test.organization"></a>Organization</h3></div></div></div><p>The directory <span class="emphasis"><em>libsrcdir/testsuite</em></span> contains theindividual test cases organized in sub-directories corresponding tochapters of the C++ standard (detailed below), the dejagnu testharness support files, and sources to various testsuite utilitiesthat are packaged in a separate testing library.</p><p>All test cases for functionality required by the runtime componentsof the C++ standard (ISO 14882) are files within the followingdirectories.</p><pre class="programlisting">17_intro18_support19_diagnostics20_util21_strings22_locale23_containers25_algorithms26_numerics27_io</pre><p>In addition, the following directories include test files:</p><pre class="programlisting">tr1 Tests for components as described by the Technical Report on Standard Library Extensions (TR1).backward Tests for backwards compatibility and deprecated features.demangle Tests for __cxa_demangle, the IA 64 C++ ABI demanglerext Tests for extensions.performance Tests for performance analysis, and performance regressions.thread Tests for threads.</pre><p>Some directories don't have test files, but instead containauxiliary information (<a class="ulink" href="#internals" target="_top">more information</a>):</p><pre class="programlisting">config Files for the dejagnu test harness.lib Files for the dejagnu test harness.libstdc++* Files for the dejagnu test harness.data Sample text files for testing input and output.util Files for libtestc++, utilities and testing routines.</pre><p>Within a directory that includes test files, there may beadditional subdirectories, or files. Originally, test caseswere appended to one file that represented a particular sectionof the chapter under test, and was named accordingly. Forinstance, to test items related to <code class="code"> 21.3.6.1 -basic_string::find [lib.string::find]</code> in the standard,the following was used:</p><pre class="programlisting">21_strings/find.cc</pre><p>However, that practice soon became a liability as the test casesbecame huge and unwieldy, and testing new or extendedfunctionality (like wide characters or named locales) becamefrustrating, leading to aggressive pruning of test cases on someplatforms that covered up implementation errors. Now, the testsuite has a policy of one file, one test case, which solves theabove issues and gives finer grained results and more manageableerror debugging. As an example, the test case quoted abovebecomes:</p><pre class="programlisting">21_strings/basic_string/find/char/1.cc21_strings/basic_string/find/char/2.cc21_strings/basic_string/find/char/3.cc21_strings/basic_string/find/wchar_t/1.cc21_strings/basic_string/find/wchar_t/2.cc21_strings/basic_string/find/wchar_t/3.cc</pre><p>All new tests should be written with the policy of one testcase, one file in mind.</p></div><div class="sect2" lang="en" xml:lang="en"><div class="titlepage"><div><div><h3 class="title"><a id="test.naming"></a>Naming Conventions</h3></div></div></div><p></p><p>In addition, there are some special names and suffixes that areused within the testsuite to designate particular kinds oftests.</p><div class="itemizedlist"><ul type="disc"><li><p><span class="emphasis"><em>_xin.cc</em></span></p><p>This test case expects some kind of interactive input in orderto finish or pass. At the moment, the interactive tests are notrun by default. Instead, they are run by hand, like:</p><pre class="programlisting">g++ 27_io/objects/char/3_xin.cccat 27_io/objects/char/3_xin.in | a.out</pre></li><li><p><span class="emphasis"><em>.in</em></span></p><p>This file contains the expected input for the corresponding <span class="emphasis"><em>_xin.cc</em></span> test case.</p></li><li><p><span class="emphasis"><em>_neg.cc</em></span></p><p>This test case is expected to fail: it's a negative test. At themoment, these are almost always compile time errors.</p></li><li><p><span class="emphasis"><em>char</em></span></p><p>This can either be a directory name or part of a longer filename, and indicates that this file, or the files within thisdirectory are testing the <code class="code">char</code> instantiation of atemplate.</p></li><li><p><span class="emphasis"><em>wchar_t</em></span></p><p>This can either be a directory name or part of a longer filename, and indicates that this file, or the files within thisdirectory are testing the <code class="code">wchar_t</code> instantiation ofa template. Some hosts do not support <code class="code">wchar_t</code>functionality, so for these targets, all of these tests will notbe run.</p></li><li><p><span class="emphasis"><em>thread</em></span></p><p>This can either be a directory name or part of a longer filename, and indicates that this file, or the files within thisdirectory are testing situations where multiple threads arebeing used.</p></li><li><p><span class="emphasis"><em>performance</em></span></p><p>This can either be an enclosing directory name or part of aspecific file name. This indicates a test that is used toanalyze runtime performance, for performance regression testing,or for other optimization related analysis. At the moment, thesetest cases are not run by default.</p></li></ul></div></div><div class="sect2" lang="en" xml:lang="en"><div class="titlepage"><div><div><h3 class="title"><a id="test.utils"></a>Utilities</h3></div></div></div><p></p><p>The testsuite directory also contains some files that implementfunctionality that is intended to make writing test cases easier,or to avoid duplication, or to provide error checking in a way thatis consistent across platforms and test harnesses. A stand-aloneexecutable, called <span class="emphasis"><em>abi_check</em></span>, and a staticlibrary called <span class="emphasis"><em>libtestc++</em></span> areconstructed. Both of these items are not installed, and only usedduring testing.</p><p>These files include the following functionality:</p><div class="itemizedlist"><ul type="disc"><li><p><span class="emphasis"><em>testsuite_abi.h</em></span>,<span class="emphasis"><em>testsuite_abi.cc</em></span>,<span class="emphasis"><em>testsuite_abi_check.cc</em></span></p><p>Creates the executable <span class="emphasis"><em>abi_check</em></span>.Used to check correctness of symbol versioning, visibility ofexported symbols, and compatibility on symbols in the sharedlibrary, for hosts that support this feature. More informationcan be found in the ABI documentation <a class="ulink" href="abi.html" target="_top">here</a></p></li><li><p><span class="emphasis"><em>testsuite_allocator.h</em></span>,<span class="emphasis"><em>testsuite_allocator.cc</em></span></p><p>Contains specialized allocators that keep track of constructionand destruction. Also, support for overriding global new anddelete operators, including verification that new and deleteare called during execution, and that allocation over max_sizefails.</p></li><li><p><span class="emphasis"><em>testsuite_character.h</em></span></p><p>Contains <code class="code">std::char_traits</code> and<code class="code">std::codecvt</code> specializations for a user-definedPOD.</p></li><li><p><span class="emphasis"><em>testsuite_hooks.h</em></span>,<span class="emphasis"><em>testsuite_hooks.cc</em></span></p><p>A large number of utilities, including:</p><div class="itemizedlist"><ul type="circle"><li><p>VERIFY</p></li><li><p>set_memory_limits</p></li><li><p>verify_demangle</p></li><li><p>run_tests_wrapped_locale</p></li><li><p>run_tests_wrapped_env</p></li><li><p>try_named_locale</p></li><li><p>try_mkfifo</p></li><li><p>func_callback</p></li><li><p>counter</p></li><li><p>copy_tracker</p></li><li><p>copy_constructor</p></li><li><p>assignment_operator</p></li><li><p>destructor</p></li><li><p>pod_char, pod_int and associated char_traits specializations</p></li></ul></div></li><li><p><span class="emphasis"><em>testsuite_io.h</em></span></p><p>Error, exception, and constraint checking for<code class="code">std::streambuf, std::basic_stringbuf, std::basic_filebuf</code>.</p></li><li><p><span class="emphasis"><em>testsuite_iterators.h</em></span></p><p>Wrappers for various iterators.</p></li><li><p><span class="emphasis"><em>testsuite_performance.h</em></span></p><p>A number of class abstractions for performance counters, andreporting functions including:</p><div class="itemizedlist"><ul type="circle"><li><p>time_counter</p></li><li><p>resource_counter</p></li><li><p>report_performance</p></li></ul></div></li></ul></div></div><div class="sect2" lang="en" xml:lang="en"><div class="titlepage"><div><div><h3 class="title"><a id="test.run"></a>Running the Testsuite</h3></div></div></div><div class="sect3" lang="en" xml:lang="en"><div class="titlepage"><div><div><h4 class="title"><a id="test.run.basic"></a>Basic Results</h4></div></div></div><p>There are several options for running tests, including testingthe regression tests, testing a subset of the regression tests,testing the performance tests, testing just compilation, testinginstalled tools, etc. In addition, there is a special rule forchecking the exported symbols of the shared library.</p><p>You can check the status of the build without installing itusing the dejagnu harness, much like the rest of the gcctools.</p><pre class="programlisting"> make check</pre><p>in the <span class="emphasis"><em>libbuilddir</em></span> directory.</p><p>or</p><pre class="programlisting"> make check-target-libstdc++-v3</pre><p>in the <span class="emphasis"><em>gccbuilddir</em></span> directory.</p><p>These commands are functionally equivalent and will create a'testsuite' directory underneath<span class="emphasis"><em>libbuilddir</em></span> containing the results of thetests. Two results files will be generated: <span class="emphasis"><em>libstdc++.sum</em></span>, which is a PASS/FAIL summary for eachtest, and <span class="emphasis"><em>libstdc++.log</em></span> which is a log ofthe exact command line passed to the compiler, the compileroutput, and the executable output (if any).</p><p>Archives of test results for various versions and platforms areavailable on the GCC website in the <a class="ulink" href="http://gcc.gnu.org/gcc-4.1/buildstat.html" target="_top">buildstatus</a> section of each individual release, and are alsoarchived on a daily basis on the <a class="ulink" href="http://gcc.gnu.org/ml/gcc-testresults/current" target="_top">gcc-testresults</a>mailing list. Please check either of these places for a similarcombination of source version, operating system, and host CPU.</p></div><div class="sect3" lang="en" xml:lang="en"><div class="titlepage"><div><div><h4 class="title"><a id="test.run.options"></a>Options</h4></div></div></div><p>To debug the dejagnu test harness during runs, try invoking with aspecific argument to the variable RUNTESTFLAGS, as below.</p><pre class="programlisting">make check-target-libstdc++-v3 RUNTESTFLAGS="-v"</pre><p>or</p><pre class="programlisting">make check-target-libstdc++-v3 RUNTESTFLAGS="-v -v"</pre><p>To run a subset of the library tests, you will need to generatethe <span class="emphasis"><em>testsuite_files</em></span> file by running<span class="command"><strong>make testsuite_files</strong></span> in the<span class="emphasis"><em>libbuilddir/testsuite</em></span> directory, describedbelow. Edit the file to remove the tests you don't want andthen run the testsuite as normal.</p><p>There are two ways to run on a simulator: set up DEJAGNU to point to aspecially crafted site.exp, or pass down --target_board flags.</p><p>Example flags to pass down for various embedded builds are as follows:</p><pre class="programlisting">--target=powerpc-eabism (libgloss/sim)make check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=powerpc-sim"--target=calmrisc32 (libgloss/sid)make check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=calmrisc32-sid"--target=xscale-elf (newlib/sim)make check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=arm-sim"</pre><p>Also, here is an example of how to run the libstdc++ testsuitefor a multilibed build directory with different ABI settings:</p><pre class="programlisting">make check-target-libstdc++-v3 RUNTESTFLAGS='--target_board \"unix{-mabi=32,,-mabi=64}\"'</pre><p>You can run the tests with a compiler and library that havealready been installed. Make sure that the compiler (e.g.,<code class="code">g++</code>) is in your <code class="code">PATH</code>. If you areusing shared libraries, then you must also ensure that thedirectory containing the shared version of libstdc++ is in your<code class="code">LD_LIBRARY_PATH</code>, or equivalent. If your GCC sourcetree is at <code class="code">/path/to/gcc</code>, then you can run the testsas follows:</p><pre class="programlisting">runtest --tool libstdc++ --srcdir=/path/to/gcc/libstdc++-v3/testsuite</pre><p>The testsuite will create a number of files in the directory inwhich you run this command,. Some of those files might use thesame name as files created by other testsuites (like the onesfor GCC and G++), so you should not try to run all thetestsuites in parallel from the same directory.</p><p>In addition, there are some testing options that are mostly ofinterest to library maintainers and system integrators. As such,these tests may not work on all cpu and host combinations, andmay need to be executed in the<span class="emphasis"><em>libbuilddir/testsuite</em></span> directory. Theseoptions include, but are not necessarily limited to, thefollowing:</p><pre class="programlisting">make testsuite_files</pre><p>Five files are generated that determine what test filesare run. These files are:</p><div class="itemizedlist"><ul type="disc"><li><p><span class="emphasis"><em>testsuite_files</em></span></p><p>This is a list of all the test cases that will be run. Eachtest case is on a separate line, given with an absolute pathfrom the <span class="emphasis"><em>libsrcdir/testsuite</em></span> directory.</p></li><li><p><span class="emphasis"><em>testsuite_files_interactive</em></span></p><p>This is a list of all the interactive test cases, using thesame format as the file list above. These tests are not runby default.</p></li><li><p><span class="emphasis"><em>testsuite_files_performance</em></span></p><p>This is a list of all the performance test cases, using thesame format as the file list above. These tests are not runby default.</p></li><li><p><span class="emphasis"><em>testsuite_thread</em></span></p><p>This file indicates that the host system can run tests whichinvolved multiple threads.</p></li><li><p><span class="emphasis"><em>testsuite_wchar_t</em></span></p><p>This file indicates that the host system can run the wchar_ttests, and corresponds to the macro definition <code class="code">_GLIBCXX_USE_WCHAR_T</code> in the file c++config.h.</p></li></ul></div><pre class="programlisting">make check-abi</pre><p>The library ABI can be tested. This involves testing the sharedlibrary against an ABI-defining previous version of symbolexports.</p><pre class="programlisting">make check-compile</pre><p>This rule compiles, but does not link or execute, the<span class="emphasis"><em>testsuite_files</em></span> test cases and displays theoutput on stdout.</p><pre class="programlisting">make check-performance</pre><p>This rule runs through the<span class="emphasis"><em>testsuite_files_performance</em></span> test cases andcollects information for performance analysis and can be used tospot performance regressions. Various timing information iscollected, as well as number of hard page faults, and memoryused. This is not run by default, and the implementation is influx.</p><p>We are interested in any strange failures of the testsuite;please email the main libstdc++ mailing list if you seesomething odd or have questions.</p></div><div class="sect3" lang="en" xml:lang="en"><div class="titlepage"><div><div><h4 class="title"><a id="test.run.permutations"></a>Test Permutations</h4></div></div></div><p>To run the libstdc++ test suite under the <a class="link" href="debug_mode.html" title="Chapter 30. Debug Mode">debug mode</a>, edit<code class="filename">libstdc++-v3/scripts/testsuite_flags</code> to add thecompile-time flag <code class="constant">-D_GLIBCXX_DEBUG</code> to theresult printed by the <code class="literal">--build-cxx</code>option. Additionally, add the<code class="constant">-D_GLIBCXX_DEBUG_PEDANTIC</code> flag to turn onpedantic checking. The libstdc++ test suite should produceprecisely the same results under debug mode that it does underrelease mode: any deviation indicates an error in either thelibrary or the test suite.</p><p>Or, just run the testsuites with <code class="constant">CXXFLAGS</code>set to <code class="constant">-D_GLIBCXX_DEBUG</code>.</p></div></div><div class="sect2" lang="en" xml:lang="en"><div class="titlepage"><div><div><h3 class="title"><a id="test.new_tests"></a>New Test Cases</h3></div></div></div><p>The first step in making a new test case is to choose the correctdirectory and file name, given the organization as previouslydescribed.</p><p>All files are copyright the FSF, and GPL'd: this is veryimportant. The first copyright year should correspond to the datethe file was checked in to SVN.</p><p>As per the dejagnu instructions, always return 0 from main toindicate success.</p><p>A bunch of utility functions and classes have already beenabstracted out into the testsuite utility library, <code class="code">libtestc++</code>. To use this functionality, just include theappropriate header file: the library or specific object files willautomatically be linked in as part of the testsuite run.</p><p>For a test that needs to take advantage of the dejagnu testharness, what follows below is a list of special keyword thatharness uses. Basically, a test case contains dg-keywords (seedg.exp) indicating what to do and what kinds of behavior are to beexpected. New test cases should be written with the new styleDejaGnu framework in mind.</p><p>To ease transition, here is the list of dg-keyword documentationlifted from dg.exp.</p><pre class="programlisting"># The currently supported options are:## dg-prms-id N# set prms_id to N## dg-options "options ..." [{ target selector }]# specify special options to pass to the tool (eg: compiler)## dg-do do-what-keyword [{ target/xfail selector }]# `do-what-keyword' is tool specific and is passed unchanged to# ${tool}-dg-test. An example is gcc where `keyword' can be any of:# preprocess|compile|assemble|link|run# and will do one of: produce a .i, produce a .s, produce a .o,# produce an a.out, or produce an a.out and run it (the default is# compile).## dg-error regexp comment [{ target/xfail selector } [{.|0|linenum}]]# indicate an error message <regexp> is expected on this line# (the test fails if it doesn't occur)# Linenum=0 for general tool messages (eg: -V arg missing).# "." means the current line.## dg-warning regexp comment [{ target/xfail selector } [{.|0|linenum}]]# indicate a warning message <regexp> is expected on this line# (the test fails if it doesn't occur)## dg-bogus regexp comment [{ target/xfail selector } [{.|0|linenum}]]# indicate a bogus error message <regexp> use to occur here# (the test fails if it does occur)## dg-build regexp comment [{ target/xfail selector }]# indicate the build use to fail for some reason# (errors covered here include bad assembler generated, tool crashes,# and link failures)# (the test fails if it does occur)## dg-excess-errors comment [{ target/xfail selector }]# indicate excess errors are expected (any line)# (this should only be used sparingly and temporarily)## dg-output regexp [{ target selector }]# indicate the expected output of the program is <regexp># (there may be multiple occurrences of this, they are concatenated)## dg-final { tcl code }# add some tcl code to be run at the end# (there may be multiple occurrences of this, they are concatenated)# (unbalanced braces must be \-escaped)## "{ target selector }" is a list of expressions that determine whether the# test succeeds or fails for a particular target, or in some cases whether the# option applies for a particular target. If the case of `dg-do' it specifies# whether the test case is even attempted on the specified target.## The target selector is always optional. The format is one of:## { xfail *-*-* ... } - the test is expected to fail for the given targets# { target *-*-* ... } - the option only applies to the given targets## At least one target must be specified, use *-*-* for "all targets".# At present it is not possible to specify both `xfail' and `target'.# "native" may be used in place of "*-*-*".Example 1: Testing compilation only// { dg-do compile }Example 2: Testing for expected warnings on line 36, which all targets fail// { dg-warning "string literals" "" { xfail *-*-* } 36Example 3: Testing for expected warnings on line 36// { dg-warning "string literals" "" { target *-*-* } 36Example 4: Testing for compilation errors on line 41// { dg-do compile }// { dg-error "no match for" "" { target *-*-* } 41 }Example 5: Testing with special command line settings, or without theuse of pre-compiled headers, in particular the stdc++.h.gch file. Anyoptions here will override the DEFAULT_CXXFLAGS and PCH_CXXFLAGS setup in the normal.exp file.// { dg-options "-O0" { target *-*-* } }</pre><p>More examples can be found in the libstdc++-v3/testsuite/*/*.cc files.</p></div><div class="sect2" lang="en" xml:lang="en"><div class="titlepage"><div><div><h3 class="title"><a id="test.dejagnu"></a>Test Harness Details</h3></div></div></div><p>Underlying details of testing are abstracted via the GNU Dejagnu package.</p><p>This is information for those looking at making changes to the testsuitestructure, and/or needing to trace dejagnu's actions with --verbose. Thiswill not be useful to people who are "merely" adding new tests to the existingstructure.</p><p>The first key point when working with dejagnu is the idea of a "tool".Files, directories, and functions are all implicitly used when they arenamed after the tool in use. Here, the tool will always be "libstdc++".</p><p>The <code class="code">lib</code> subdir contains support routines. The<code class="code">lib/libstdc++.exp</code> file ("support library") is loadedautomagically, and must explicitly load the others. For example, files canbe copied from the core compiler's support directory into <code class="code">lib</code>.</p><p>Some routines in <code class="code">lib/libstdc++.exp</code> are callbacks, some areour own. Callbacks must be prefixed with the name of the tool. To easilydistinguish the others, by convention our own routines are named "v3-*".</p><p>The next key point when working with dejagnu is "test files". Anydirectory whose name starts with the tool name will be searched for test files.(We have only one.) In those directories, any <code class="code">.exp</code> file isconsidered a test file, and will be run in turn. Our main test file is called<code class="code">normal.exp</code>; it runs all the tests in testsuite_files using thecallbacks loaded from the support library.</p><p>The <code class="code">config</code> directory is searched for any particular "targetboard" information unique to this library. This is currently unused and setsonly default variables.</p></div><div class="sect2" lang="en" xml:lang="en"><div class="titlepage"><div><div><h3 class="title"><a id="test.future"></a>Future</h3></div></div></div><p></p><p>Shared runs need to be implemented, for targets that support shared libraries.</p><p>Diffing of expected output to standard streams needs to be finished off.</p><p>The V3 testing framework supports, or will eventually support,additional keywords for the purpose of easing the job of writingtest cases. All V3-keywords are of the form <code class="code">@xxx@</code>.Currently plans for supported keywords include:</p><div class="variablelist"><dl><dt><span class="term"> <code class="code"> @require@ <files> </code> </span></dt><dd><p>The existence of <files> is essential for the test to completesuccessfully. For example, a test case foo.C using bar.baz asinput file could say</p><pre class="programlisting">// @require@ bar.baz</pre><p>The special variable % stands for the rootname, e.g. thefile-name without its `.C' extension. Example of use (takenverbatim from 27_io/filebuf.cc)</p><pre class="programlisting">// @require@ %-*.tst %-*.txt</pre></dd><dt><span class="term"> <code class="code"> @diff@ <first-list> <second-list> </code> </span></dt><dd><p>After the test case compiles and ran successfully, diff<first-list> against <second-list>, these lists shouldhave the same length. The test fails if diff returns non-zero apair of files.</p></dd></dl></div></div></div><div class="navfooter"><hr /><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"><a accesskey="p" href="build.html">Prev</a> </td><td width="20%" align="center"><a accesskey="u" href="bk01pt01ch02.html">Up</a></td><td width="40%" align="right"> <a accesskey="n" href="using.html">Next</a></td></tr><tr><td width="40%" align="left" valign="top">Build </td><td width="20%" align="center"><a accesskey="h" href="../spine.html">Home</a></td><td width="40%" align="right" valign="top"> Chapter 3. Using</td></tr></table></div></body></html>