Last updated: 11-Jun-2006 jh/ya
To check for possible updates to this document, please see http://www.spec.org/cpu2006/Docs/
Contents
Editing result description fields
Generating a rate result from a speed result
This document describes various useful utilities that ship with SPEC CPU2006. It is assumed that you have already read runspec.html and config.html.
Note: links to SPEC CPU2006 documents on this web page assume that you are reading the page from a directory that also contains the other SPEC CPU2006 documents. If by some chance you are reading this web page from a location where the links do not work, try accessing the referenced documents at one of the following locations:
Most of the examples in this document have been written in Unix syntax. That means you'll need to use your imagination just a bit as you read this document:
| If you see: | Just pretend you saw: |
| cat | type |
|---|---|
| cp | copy |
| diff | fc |
| grep | findstr |
| ls | dir |
| / | \ |
| $SPEC | %spec% |
Also, some of the scripts cannot be invoked directly, but instead must be invoked by putting
specperl %spec%\bin\
in front of the script name. This is noted in the sections where it is needed.
"configpp" is a short way to say:
$ runspec --action configpp
That switch is mentioned in runspec.html but you'll find much more about the config file preprocessor in config.html. The configuration file is specified using the --config switch, or -c for short, at the command line. For example, the following two command are equivalent:
$ runspec --action configpp -c try_utilities.cfg $ configpp -c try_utilities.cfg
There are a couple of points worth mentioning here:
The configpp utility is a good way to check your configuration file for fatal errors, as the config file is completely parsed before it's output.
The configpp utility is new with SPEC CPU2006.
If you have a rawfile, you can extract the config file that was used with it. To do so, use the extract_config utility.
For example, starting with a freshly-installed SPEC CPU2006 directory tree:
% runspec --config=try_utilities --size=test --iterations=1 \ --tuning=base --output_format=text,raw bzip2 > /dev/null % cd result % ls CINT2006.001.test.rsf CINT2006.001.test.txt CPU2006.001.log CPU2006.lock
Note that we have just the single result. Let's make a temporary directory and copy the rawfile there:
% mkdir tmp % cd tmp % cp ../CINT2006.001.test.rsf . % ls CINT2006.001.test.rsf
(On Windows, that ls command would be spelt dir. You'll find some more Windows notes in the introduction.)
Now, attempt to extract the config file:
% extract_config CINT2006.001.test.rsf > tmp.cfg
Does the extracted file match the original?
% diff -u ../../config/try_utilities.cfg tmp.cfg
--- ../../config/try_utilities.cfg 2006-05-22 14:41:42.000000000 -0700
+++ tmp.cfg 2006-05-22 14:43:16.000000000 -0700
@@ -1,3 +1,6 @@
+# Invocation command line:
+# /Volumes/storage/WorkRelated/SPEC/CPU2006/Kits/cpu2006-kit91/bin/runspec
--config=try_utilities --size=test --iterations=1 --tuning=base
--output_format=asc,raw bzip2
+############################################################################
# This is a sample config file. It was tested with:
#
# Compiler name/version: Apple GCC v4.0.1 build 5245
@@ -128,15 +131,8 @@
sw_state = Default
sw_other = None
-__MD5__
-401.bzip2=base=try_utilities=default:
-# Last updated Mon May 22 14:41:42 2006
-optmd5=0666e60587f33e2aa13eb6ba0917aa60
-baggage=
-compile_options=\
-@eNqNj00LgkAQhu/7K4a9b5duoUKuJpa6C+qlS9QmYqAjaYf+faMF2cfBYQ8Dsy/P80oV6xWUxoCg\
-h4BtX2HTLRCEl2pfHqTOaU08380DoBFqCaI+35tjXRnRoGgrCpbwNVaHt6spHCZXIKXNicDHXblb\
-pTObf+A4IwkibaJ1kNLtF82ZGsI6C+Nw79OX/xqcRWGyexaa4Wrh6VKYvnMm5m+rwTfyXu4z8WNE\
-5UPBSbsHPPNh9Q==
-exemd5=d828496b92983e96408f680b55393225
%
Yes. The only differences are:
Instead of saying diff just above, you would say fc (and, as already mentioned, dir is the Windows spelling for what Unix calls ls. You can invoke extract_config as:
specperl %spec%\bin\extract_config rawfile
If you have a pdf, html, or ps output file, you can extract the rawfile that was used to generate it. To do so, use extract_raw.
For example:
% runspec --config=try_utilities --size=test --iterations=1 \ --tuning=base --output_format=ps bzip2 > /dev/null % cd $SPEC/result % ls CINT2006.001.test.rsf CINT2006.001.test.ps CPU2006.001.log CPU2006.lock % mkdir tmp % cd tmp % cp ../*ps . % ls CINT2006.001.test.ps
Note that at this point we have just the postscript output in the current directory. Now, attempt to extract the raw file:
% extract_raw *ps Wrote "CINT2006.001.test.rsf" % ls CINT2006.001.test.ps CINT2006.001.test.rsf
Does the extracted file match the original?
% diff -u ../CPU2006.001.rsf ./CPU2006.001.rsf %
Yes.
You can invoke extract_raw as:
specperl %spec%\bin\extract_raw file-to-extract-from
Note that the file you extract from can be pdf, ps, or html, just as on Unix systems, with one important exception: if you are using a browser that attempts to add value to the html when you save it, that browser may break the embedded rawfile. For example, with Internet Explorer Version 5.00.3103.1000, if you are viewing a SPEC CPU2006 html file and you select File -> Save As, the default for "Save as type" is shown as
"Web Page, complete (*.htm,*.html)"
Do not accept that default if you want to be able to retrieve the rawfile later on. If you take the default, your saved file will be incomplete. To save the web page INCLUDING the embedded rawfile, you must change "Save as type" to
"Web Page, HTML only (*.htm,*.html)"
If you have a rawfile, you can extract the XML flags description file that the result uses. To do so, use extract_flags. Usage is identical to extract_raw, but unlike extract_raw, may not produce an flags description file identical to the input. This is because EOLs (end-of-line) characters are normalized to Unix-style LF (line feed) only.
Note: If you have used a flags description file to format your result, the entire flags file is stored in the raw file (not just the part needed for result).
For example, if the flags description file "flags-macosx-ia32-iccifortv91.xml" was used to format the CFP2006.002.rsf raw file, then you can extract the original flags file using the following command
% pwd
/cpu2006/cpu2006-kit92/result
% ls
CFP2006.002.rsf images
% extract_flags CFP2006.002.rsf > tmp.xml
% ls ../*xml
../flags-macosx-ia32-iccifortv91.xml
% diff -u ../flags-macosx-ia32-iccifortv91.xml tmp.xml
%
We see absolutely no difference between the extraced flags description file "tmp.xml" and the original flags description file "flags-macosx-ia32-iccifortv91.xml"
Note: If a raw file has not been formatted with a flags description file, extract_flags does nothing.
The extract_flags utility is new with CPU2006.
If you have an XML flag description file, such as one extracted using extract_flags, or one downloaded from SPEC, flag_dump will create a more human-readable HTML version from it.
For example:
% cd $SPEC/Docs
% ls *xml
flags-advanced.xml flags-simple.xml
% cp flags-advanced.xml willi_flags.xml
% flag_dump willi_flags.xml
Processing willi_flags.xml...wrote willi_flags.html
% ls willi_flags.*
willi_flags.html willi_flags.xml
Note that flag_dump utility assumes that you are formatting flags files written by users. These files are not allowed to contain mandatory flags. If you attempt to use the flag_dump utility to format any of the suite-provided flag files such as flags_mandatory.xml or the benchmark flag files that reside in $SPEC/benchspec/CPU2006/*/Spec/ directory, the utility will exit with an error message such as the one below.
% cd $SPEC/benchspec
% flag_dump flags_mandatory.xml
Processing flags_mandatory.xml...
ERROR: The flag file at
flags_mandatory.xml
contains one or more flags in the class "mandatory", which is not
allowed.
%
The flag_dump utility is new with CPU2006.
When you source the shrc, as described in runspec.html, the 'go' shell alias is created. go is a handy utility to navigate the directory structure of a SPEC source tree. Here are the various things it understands and the places they will take you:
| If you say | You'll end up in |
| go
go top | $SPEC |
|---|---|
| go bin | $SPEC/bin (where the tools live) |
| go config | $SPEC/config |
| go result
go results | $SPEC/result |
| go doc
go docs | $SPEC/Docs |
| go cpu | $SPEC/benchspec/CPU2006 |
| go benchmark | The top level directory for the named benchmark, if it can be found. (Examples follow) |
| go 401 | $SPEC/benchspec/CPU2006/401.bzip2 |
| go perlb | $SPEC/benchspec/CPU2006/400.perlbench Notice that abbreviated names are accepted. But if the abbreviation is ambiguous, the command will pick the first match, so pay attention to the reply: $ go g /spec/cpu2006/kit91/benchspec/CPU2006/403.gcc $ go go /spec/cpu2006/kit91/benchspec/CPU2006/445.gobmk $ |
| go nada | no change: there's no benchmark named nada* |
| go benchmark subdir | The named subdirectory of the directory tree for the referenced benchmark, for example "go perlb run". You can also name more than one level of subdirectory: "go 462 data train". The go alias existed in CPU2000, but for CPU2006 the order is new when naming subdirectories, as is the ability to name more than one level of subdirectory. |
| go subdir | If you are already in a benchmark subtree, then you can move to a different branch of the same benchmark, by naming the branch and not mentioning a benchmark. For example: $ pwd /rahul/benchspec/CPU2006/401.bzip2/run/build_base_a3.0000 $ go exe /rahul/benchspec/CPU2006/401.bzip2/exe $ |
Note: If you leave the shell where you sourced shrc, the alias is no longer available! So if you want to use this alias, do not start a sub-shell.
Note: 'go' does not know about your settings for output_root or expid, and will not be able to take you to your run, exe, or result directories if you use those features. See ogo for something that can help.
New with CPU2006, if you use a csh-like shell, please note that after you say source cshrc, there is a go alias that is set up for you, which implements the same features as described above.
Sorry, the go feature does not exist for Windows systems.
When you source shrc, as described in runspec.html, the 'ogo' shell alias is created. Its function is similar to the go alias with one important difference: if you use the output_root setting in your config file (also discussed in runspec.html) you can use ogo as a shorthand for moving around both the original installed benchmark tree and your output tree.
To do so, simply set the GO environment variable to the same value as your output_root. Then, 'ogo' will use the contents of that variable as the top level of the benchmark tree when looking for run directories and results.
For example, you could say something like this:
$ go config /spec/config $ grep output_root cds.cfg output_root = /cds_stuff $ GO=/cds_stuff $ ogo run perl Using value in GO for output_root: /cds_stuff /cds_stuff/benchspec/CPU2006/400.perlbench/run $
In this example, the benchmarks are installed in "/spec", and output_root is set to "/cds_stuff" in the config file. Any runs using that config file will write their results, executables, and run directories under "/cds_stuff". Continuing with other examples:
| If you say | You'll end up in |
| ogo result | /cds_stuff/result
Your own private result directory |
|---|---|
| ogo 400 run | /cds_stuff/benchspec/CPU2006/400.perlbench/run
Your own private run directories |
| ogo bzip exe | /cds_stuff/benchspec/CPU2006/401.bzip2/exe
Your own private executables |
| ogo doc | /spec/Docs
There's only one Docs directory, on the original installed $SPEC tree |
| ogo config | /spec/config
The config directory is always on the original installed tree; set its protections as described in runspec.html |
| ogo 400 src | /spec/benchspec/CPU2006/400.perlbench/src
Run directories are under output_root, but the benchmark src directory is still in the original installed $SPEC tree. |
Note: It's not necessary to export the "GO" variable. Declaring this variable at the command-prompt is sufficient.
$ GO=/tmp/cds
$ echo $GO
/tmp/cds
The ogo alias is new with CPU2006.
ogo has also been implemented for cshrc. There is one restriction: to get the full feature set above, you must set OGO to the actual directory, not to a symbolic link.
Sorry, the ogo feature does not exist for Windows systems.
Some benchmark executables are invoked multiple times. If you'd like to know how much time was spent on each invocation, it may not be easy to tell. You could examine "speccmds.out", but that file is not easy to interpret:
$ runspec --config=try_utilities --size=test --iterations=1 \
--tuning=base --output_format=ps 400
$ go work 400
$ cd run_base_test_try_utilities.0000/
$
$ cat speccmds.out | fold -s
running commands in speccmds.cmd 1 times
runs started at 1148417981, 220153000, Tue May 23 13:59:41 2006
run 1 started at 1148417981, 222495000, Tue May 23 13:59:41 2006
child started: 0, 1148417981, 222535000, pid=2396,
'../run_base_test_try_utilities.0000/perlbench_base.try_utilities -I. -I./lib
attrs.pl'
child finished: 0, 1148417981, 339666000, sec=0, nsec=117131000, pid=2396, rc=0
child started: 0, 1148417981, 339754000, pid=2397,
'../run_base_test_try_utilities.0000/perlbench_base.try_utilities -I. -I./lib
gv.pl'
child finished: 0, 1148417981, 384163000, sec=0, nsec=44409000, pid=2397, rc=0
child started: 0, 1148417981, 384240000, pid=2398,
'../run_base_test_try_utilities.0000/perlbench_base.try_utilities -I. -I./lib
makerand.pl'
child finished: 0, 1148417981, 710682000, sec=0, nsec=326442000, pid=2398, rc=0
child started: 0, 1148417981, 710786000, pid=2399,
'../run_base_test_try_utilities.0000/perlbench_base.try_utilities -I. -I./lib
pack.pl'
child finished: 0, 1148417981, 973602000, sec=0, nsec=262816000, pid=2399, rc=0
child started: 0, 1148417981, 973684000, pid=2400,
'../run_base_test_try_utilities.0000/perlbench_base.try_utilities -I. -I./lib
redef.pl'
child finished: 0, 1148417981, 997347000, sec=0, nsec=23663000, pid=2400, rc=0
child started: 0, 1148417981, 997429000, pid=2401,
'../run_base_test_try_utilities.0000/perlbench_base.try_utilities -I. -I./lib
ref.pl'
child finished: 0, 1148417982, 24001000, sec=0, nsec=26572000, pid=2401, rc=0
child started: 0, 1148417982, 24081000, pid=2402,
'../run_base_test_try_utilities.0000/perlbench_base.try_utilities -I. -I./lib
regmesg.pl'
child finished: 0, 1148417982, 55465000, sec=0, nsec=31384000, pid=2402, rc=0
child started: 0, 1148417982, 55621000, pid=2403,
'../run_base_test_try_utilities.0000/perlbench_base.try_utilities -I. -I./lib
test.pl'
child finished: 0, 1148417988, 281976000, sec=6, nsec=226355000, pid=2403, rc=0
run 1 finished at: 1148417988, 282030000, Tue May 23 13:59:48 2006
run 1 elapsed time: 7, 59535000, 7.059535000
runs finished at 1148417988, 282155000, Tue May 23 13:59:48 2006
runs elapsed time: 7, 62002000, 7.062002000
$
Fortunately, SPEC has provided a tool to interpret the above for you:
$ printkids.pl
Seconds Command
0.12 test_try_utilities.0000/perlbench_base.try_utilities -I. -I./lib attr
0.04 test_try_utilities.0000/perlbench_base.try_utilities -I. -I./lib gv.p
0.33 test_try_utilities.0000/perlbench_base.try_utilities -I. -I./lib make
0.26 test_try_utilities.0000/perlbench_base.try_utilities -I. -I./lib pack
0.02 test_try_utilities.0000/perlbench_base.try_utilities -I. -I./lib rede
0.03 test_try_utilities.0000/perlbench_base.try_utilities -I. -I./lib ref.
0.03 test_try_utilities.0000/perlbench_base.try_utilities -I. -I./lib regm
6.23 test_try_utilities.0000/perlbench_base.try_utilities -I. -I./lib test
=========
7.06 Total by adding children
7.06 Total according to speccmds.out
$
If you rearrange mount points or rename your SPEC root directory, you will probably find confusing problems like the one in the example that follows. We used to have a working SPEC tree under /cpu2006/cpu2006-kit91, but we're going to change that:
$ cd /cpu2006
$ mv cpu2006-kit91 jamesbond
$ cd jamesbond
$ . ./shrc
$ runspec -h
-bash: /cpu2006/jamesbond/bin/runspec: /cpu2006/cpu2006-kit91/bin/specperl:
bad interpreter: No such file or directory
The problem is that the first line of runspec points to a path that no longer exists:
$ head -1 bin/runspec
#!/cpu2006/cpu2006-kit91/bin/specperl
$
Fortunately, SPEC has provided a tool to relocate the tools in the new tree:
$ cd /cpu2006/jamesbond
$ . ./shrc
$ relocate
Top of SPEC benchmark tree is '/cpu2006/jamesbond'
$ runspec -h | head -3
runspec v4256 - Copyright 1999-2006 Standard Performance Evaluation Corporation
Using 'macosx-ppc' tools
Reading MANIFEST... 2565 files
Limitations: There are some limitations to this tool. Most notably, it uses perl, so if the tree has been re-arranged in such a way that perl has an identity crisis and can't find itself, you'll be in trouble. Make sure that you have sourced shrc, as in the next example; if it still fails, you may need to reinstall the CPU2006 kit.
$ cd /cpu2006
$ mv jamesbond dr_evil
$ cd dr_evil
$ bin/relocate
Your $SPEC variable is not set correctly!
Top of SPEC benchmark tree is '/cpu2006/dr_evil'
Can't locate strict.pm in @INC (@INC contains: /cpu2006/jamesbond/bin
/cpu2006/jamesbond/bin/lib .) at bin/relocate line 3.
BEGIN failed--compilation aborted at bin/relocate line 3.
$ . ./shrc
$ bin/relocate
Top of SPEC benchmark tree is '/cpu2006/dr_evil'
$ runspec -h | head -2
runspec v4256 - Copyright 1999-2006 Standard Performance Evaluation Corporation
Using 'macosx-ppc' tools
Reading MANIFEST... 2565 files
$
"rawformat" is simply another name for:
runspec --rawformat
So please see runspec.html for basic information on using this command. But a few additional points are worth mentioning:
Editing result description fields
What do you do when Marketing decides to change the name from "SuperHero 4" to "SuperHero IV" just before you publish the results, or just before you're ready to submit them to SPEC?
SPEC does allow you to edit the rawfile to make corrections. Just make sure that you only edit the fields located before the "fence":
# =============== do not edit below this point ===================
WARNING: Make a backup copy of the rawfile first!
For example:
$ cd $SPEC/result
$ cp CINT2006.001.rsf new.rsf
edit new.rsf to make your changes
$ diff CINT2006.001.rsf new.rsf
9c9
< spec.cpu2006.hw_model: SuperHero 4
---
> spec.cpu2006.hw_model: SuperHero IV
$
$ rawformat -o ps new.rsf
runspec v4256 - Copyright 1999-2006 Standard Performance Evaluation Corporation
Using 'macosx-ppc' tools
Reading MANIFEST... 2565 files
Loading runspec modules...............
Locating benchmarks...found 31 benchmarks in 2 benchsets.
Locating output formats: ASCII, config, CSV, flags, html, mail, PDF,
PostScript, raw, Screen, Submission Check
Formatting new.rsf
Parsing flags for 400.perlbench base: done
Parsing flags for 401.bzip2 base: done
Parsing flags for 403.gcc base: done
Parsing flags for 429.mcf base: done
Parsing flags for 445.gobmk base: done
Parsing flags for 456.hmmer base: done
Parsing flags for 458.sjeng base: done
Parsing flags for 462.libquantum base: done
Parsing flags for 464.h264ref base: done
Parsing flags for 471.omnetpp base: done
Parsing flags for 473.astar base: done
Parsing flags for 483.xalancbmk base: done
Doing flag reduction: done
format: PostScript -> new.ps
There is no log file for this run.
runspec finished at Tue May 23 15:09:25 2006; 16 total seconds elapsed
$
About continuation fields in rawfiles
Notice when editing fields that have more than one line that you need to construct the numbers with correct syntax. The syntax in the rawfile is a bit more picky than in config files (compare the following vs. the config.html section on "Field scoping and continuation". You cannot say something like this:
| WRONG: |
spec.cpu2006.sw_compiler: Intel C Compiler 9.1 for Windows XP
spec.cpu2006.sw_compiler1: Visual Studio 2005 (for libraries) |
because the tools are not prepared at this point for a mixture of numbered and un-numbered fields. If you need to have continued fields, say instead:
| RIGHT: |
spec.cpu2006.sw_compiler0: Intel C Compiler 9.1 for Windows XP
spec.cpu2006.sw_compiler1: Visual Studio 2005 (for libraries) |
How do you generate a rate result from a speed result?
To generate a 1-copy rate result from a speed run, copy the original rawfile to another location, and use rawformat to both generate the new rawfile and whatever other reports you want. For example:
$ cd $SPEC/result
$ grep SPECint CINT2006.003.txt
SPECint_base2006 10.1
SPECint2006 10.1
$ cp CINT2006.003.rsf convertme
$ rawformat --output_format asc,raw,ps --rate convertme
runspec v4256 - Copyright 1999-2006 Standard Performance Evaluation Corporation
Using 'macosx-ppc' tools
Reading MANIFEST... 2565 files
Loading runspec modules...............
Locating benchmarks...found 31 benchmarks in 2 benchsets.
Locating output formats: ASCII, config, CSV, flags, html, mail, PDF,
PostScript, raw, Screen, Submission Check
Formatting convertme
The version of the CPU2006 suite used to produce this result is NOT a release
version.
Parsing flags for 400.perlbench base: done
Parsing flags for 400.perlbench peak: done
Parsing flags for 401.bzip2 base: done
Parsing flags for 401.bzip2 peak: done
Parsing flags for 403.gcc base: done
Parsing flags for 403.gcc peak: done
Parsing flags for 429.mcf base: done
Parsing flags for 429.mcf peak: done
Parsing flags for 445.gobmk base: done
Parsing flags for 445.gobmk peak: done
Parsing flags for 456.hmmer base: done
Parsing flags for 456.hmmer peak: done
Parsing flags for 458.sjeng base: done
Parsing flags for 458.sjeng peak: done
Parsing flags for 462.libquantum base: done
Parsing flags for 462.libquantum peak: done
Parsing flags for 464.h264ref base: done
Parsing flags for 464.h264ref peak: done
Parsing flags for 471.omnetpp base: done
Parsing flags for 471.omnetpp peak: done
Parsing flags for 473.astar base: done
Parsing flags for 473.astar peak: done
Parsing flags for 483.xalancbmk base: done
Parsing flags for 483.xalancbmk peak: done
Doing flag reduction: done
format: raw -> convertme.rsf
format: ASCII -> convertme.txt
format: PostScript -> convertme.ps
There is no log file for this run.
runspec finished at Tue May 23 15:17:13 2006; 16 total seconds elapsed
$ grep SPECint convertme.txt
SPECint_rate_base2006 10.1
SPECint_rate2006 10.1
$
You may find that --output_format will only accept one argument at a time. So, first create the rawfile, by using --output_format raw, then use the new rawfile to create the other reports.
In order to decide whether your system got the correct answer for a benchmark, runspec writes a file called compare.cmd in the run directory, and then calls specinvoke, which calls specdiff. (The specinvoke utility is described below.)
Suppose that your results are flagged as incorrect, but you'd like to have more detail. You could ask for more than 10 lines of detail by changing the config file's default for difflines from 10 to something else, and re-running the experiment. But, you might save a lot of time and avoid the re-run of the experiment, by calling specdiff directly, with a different number of lines for its switch -l.
For example, a file chock-full of wrong answers for 444.namd has been created by hand. First, let's see how specdiff would normally be run (line wraps added for readability):
$ go run namd
$ cd run_base_test_try_utilities.0000/
$ specinvoke -c 1 -f compare.cmd -n
# Use another -n on the command line to see chdir commands
# Starting run for copy #0
specperl /cpu2006/dr_evil/bin/specdiff -m -l 10 --abstol 1e-05 \
/cpu2006/dr_evil/benchspec/CPU2006/444.namd/data/test/output/namd.out namd.out > namd.out.cmp
The above says that normally we test for correct answers by comparing the file $SPEC/benchspec/CPU2006/444.namd/data/test/output/namd.out versus the file namd.out in the current directory. Normally, 10 lines of detail are printed. Let's change that to 15 lines, and compare versus our intentionally wrong file instead:
$ specdiff -m -l 15 --abstol 1e-05 \
/cpu2006/dr_evil/benchspec/CPU2006/444.namd/data/test/output/namd.out namd_wrong.out
0003: -355359.705797 0.000000 24229.348329
-355359.705797 0.000000 23229.338329
^
0004: 229430.745102 5781.466932 -2454.694382
229330.735102 5781.366932 -2353.693382
^
0005: 5781.466932 226184.623759 -7867.535225
5781.366932 226183.623759 -7867.535225
^
0006: -2454.694382 -7867.535225 252281.168869
-2353.693382 -7867.535225 252281.168869
^
0010: 0.000000 -0.000000 0.000000 24.034240
-0.000000 -0.000000 0.000000 23.033230
^
0012: 0.000000 -0.000000 0.000000 24.034240
-0.000000 -0.000000 0.000000 23.033230
^
0016: -355359.705797 1175488.426964 24229.348329
-355359.705797 1175388.326963 23229.338329
^
0017: 229430.745102 5781.466932 -2454.694382
229330.735102 5781.366932 -2353.693382
^
0018: 5781.466932 226184.623759 -7867.535225
5781.366932 226183.623759 -7867.535225
^
0019: -2454.694382 -7867.535225 252281.168869
-2353.693382 -7867.535225 252281.168869
^
0020: 5180.324284 277.843622 -1135.837232
5180.323283 277.833622 -1135.837232
^
0021: 277.843622 2960.457709 -681.816292
277.833622 2960.357709 -681.816292
^
0022: -1135.837232 -681.816292 4515.105462
-1135.837232 -681.816292 3515.105362
^
0023: 0.000000 -0.000000 0.000000 24.034240
-0.000000 -0.000000 0.000000 23.033230
^
0024: -0.000000 -0.000000 0.000000 0.654402
0.000000 -0.000000 0.000000 0.653302
^
0025: 0.000000 -0.000000 0.000000 24.079992
-0.000000 -0.000000 0.000000 23.079992
^
specdiff run completed
$
Here are the switches supported by specdiff:
$ specdiff -h
Usage: /Users/cloyce/SPEC/cpu2006/bin/specdiff <options> file1 [file2]
-l,--lines # of lines of differences to print (-1 for all)
-q,--quiet don't print lines just set return code
-a,--abstol absolute tolerance (for floating point compares)
-r,--reltol relative tolerance (for floating point compares)
-o,--obiwan allow off-by-one errors
-O,--OBIWAN *don't* allow off-by-one errors
-s,--skiptol # of differences to ignore
--skipabstol # of abstol differences to ignore
--skipreltol # of reltol differences to ignore
--skipobiwan # of off-by-one differences to ignore
-t,--calctol calculate required tolerances
-H,--histogram show a histogram of differences
-f,--floatcompare Do comparison for floating-point values even if no
tolerances are specified. (Normally this is
automatic.)
-m,--mis write file2.mis with miscompares
-c,--cw collapse whitespace (probably doesn't do what you
think it does)
-C,--CW *don't* collapse whitespace
-b,--binary Do byte-wise comparison for binary outputs
-i,--ignorecase Ignore case for string comparisons
-d,--datadir Compare against file(s) in data directory
--os Set the operating system type (this is unnecessary)
-v,--verbose Set the level of noisiness for the output
-h,--help Print this message
$
In order to run a benchmark, runspec writes a series of commands in the run directory, in the file speccmds.cmd. These commands are then interpreted by specinvoke.
When something goes wrong, it is often useful to try the commands by hand. Ask specinvoke what it did, by using the "-n" switch:
$ go work namd
/cpu2006/dr_evil/benchspec/CPU2006/444.namd/run
$ cd run_base_test_try_utilities.0000
$ ls
compare.cmd namd.input namd_base.try_utilities
compare.err namd.out.cmp speccmds.cmd
compare.out namd.out.mis speccmds.err
compare.stdout namd.out.orig speccmds.out
namd.err namd.stdout speccmds.stdout
$ specinvoke -n
# Use another -n on the command line to see chdir commands
# Starting run for copy #0
../run_base_test_try_utilities.0000/namd_base.try_utilities \
--input namd.input --iterations 1 \
--output namd.out > namd.stdout 2>> namd.err
$
In the above example, we see how namd is invoked (line wraps added for readability). If you wanted to try out your own variations on the above command by hand, you could do so.
Here are the switches supported by specinvoke:
$ specinvoke -h
Usage: specinvoke [options] [instructionfile]
-i # iterations
-c # concurrent processes (overrides -u in command file)
-E return non-zero exit code if child does
-s shell shell to invoke
-f file instruction file
-o file output file
-e file error file
-d dir change to dir first
-n[n] print a 'dry_run' of commands
-A return 0 error code
-S msecs sleep between spawning copies (in milliseconds)
-r don't do I/O redirection ($command already has it)
-N open null device when no input file is specified
-Z use zero-length file when no input file is specified
-C (default) close stdin when no input file is specified
-h this message
To build a benchmark, runspec uses specmake, which is simply gnu make under a unique name to avoid possible conflicts with other versions of make on your system.
If a benchmark build fails, one of the most massively useful things you can do is to ask specmake what it did, by using the "-n" switch, which means "dry run".
For example, let's create a config file that is unlikely to work:
$ go config
$ cat > tmp.cfg
ext = silly
FC = f66
$
and try it out:
$ runspec --config=tmp --tuning=base --action=build leslie3d
runspec v4256 - Copyright 1999-2006 Standard Performance Evaluation Corporation
Using 'macosx-ppc' tools
Reading MANIFEST... 2565 files
Loading runspec modules...............
Locating benchmarks...found 31 benchmarks in 13 benchsets.
Locating output formats: ASCII, config, CSV, flags, html, mail, PDF,
PostScript, raw, Screen, Submission Check
Reading config file '/cpu2006/dr_evil/config/tmp.cfg'
Benchmarks selected: 437.leslie3d
Compiling Binaries
Building 437.leslie3d ref base silly default: (build_base_silly.0000)
Error with make 'specmake build': check file
'/cpu2006/dr_evil/benchspec/CPU2006/437.leslie3d/run/build_base_silly.0000/make.err'
Error with make!
*** Error building 437.leslie3d
If you wish to ignore this error, please use '-I' or ignore errors.
The log for this run is in /cpu2006/dr_evil/result/CPU2006.009.log
runspec finished at Thu May 25 10:01:54 2006; 8 total seconds elapsed
$
Sure enough, it failed. Let's find the appropriate run directory where our silly build was attempted:
$ go work leslie3d
/cpu2006/dr_evil/benchspec/CPU2006/437.leslie3d/run
$ ls
build_base_silly.0000 list
$ cd build_base_silly.0000
And ask specmake just exactly what it generated:
$ specmake -n
f66 -c -o tml.o tml.f
f66 tml.o -o leslie3d
OK, let's enter the command by hand:
$ f66 -c -o tml.o tml.f
-bash: f66: command not found
$
A great convenience of the CPU2006 suite is the fact that you can now proceed to play with all the files in this run directory, to prototype different changes and try out ideas, without fear that you will break anything. As long as you confine your edits to (in this example) $SPEC/benchspec/CPU2006/437.leslie3d/run/build_base_silly.0000, you are in a private sandbox and will not harm the original sources.
Here are the switches supported by specmake:
$ specmake -h
Usage: specmake [options] [target] ...
Options:
-b, -m Ignored for compatibility.
-B, --always-make Unconditionally make all targets.
-C DIRECTORY, --directory=DIRECTORY
Change to DIRECTORY before doing anything.
-d Print lots of debugging information.
--debug[=FLAGS] Print various types of debugging information.
-e, --environment-overrides
Environment variables override makefiles.
-f FILE, --file=FILE, --makefile=FILE
Read FILE as a makefile.
-h, --help Print this message and exit.
-i, --ignore-errors Ignore errors from commands.
-I DIRECTORY, --include-dir=DIRECTORY
Search DIRECTORY for included makefiles.
-j [N], --jobs[=N] Allow N jobs at once; infinite jobs with no arg.
-k, --keep-going Keep going when some targets can't be made.
-l [N], --load-average[=N], --max-load[=N]
Don't start multiple jobs unless load is below N.
-n, --just-print, --dry-run, --recon
Don't actually run any commands; just print them.
-o FILE, --old-file=FILE, --assume-old=FILE
Consider FILE to be very old and don't remake it.
-p, --print-data-base Print make's internal database.
-q, --question Run no commands; exit status says if up to date.
-r, --no-builtin-rules Disable the built-in implicit rules.
-R, --no-builtin-variables Disable the built-in variable settings.
-s, --silent, --quiet Don't echo commands.
-S, --no-keep-going, --stop
Turns off -k.
-t, --touch Touch targets instead of remaking them.
-v, --version Print the version number of make and exit.
-w, --print-directory Print the current directory.
--no-print-directory Turn off -w, even if it was turned on implicitly.
-W FILE, --what-if=FILE, --new-file=FILE, --assume-new=FILE
Consider FILE to be infinitely new.
--warn-undefined-variables Warn when an undefined variable is referenced.
This program built for powerpc-apple-darwin6.8
Report bugs to <bug-make@gnu.org>
If you have a rawfile, you can get information about the binary toolset and the versions of all of the system-independent tools files. To do so, use the toolsver utility.
For example, starting with a freshly-installed SPEC CPU2006 directory tree:
% runspec --config=try_utilities --size=test --iterations=1 \
--tuning=base --output_format=ps > /dev/null
% cd $SPEC/result
% ls
CINT2006.001.test.ps CINT2006.001.test.rsf CPU2006.001.log CPU2006.lock
Note that we have just the single result file. Now, attempt to extract the version information:
% toolsver CINT2006.001.test.rsf
CINT2006.001.test.rsf:
Binary toolset used was 'macosx-ppc'
Versions of Perl-based tools:
4256 runspec
4163 CPUFlagsParser.pm
4160 PSPDF.pm
4253 asc.pl
4256 benchmark.pm
4223 benchset.pm
4238 cfgfile.pl
3626 compare.pl
4243 config.pl
4247 csv.pl
4238 flags.pl
4238 flagutils.pl
3626 format.pm
4238 listfile.pm
4166 locate.pl
4160 log.pl
4214 mail.pl
3626 monitor.pl
4160 os.pl
4252 parse.pl
4160 pdf.pl
4253 ps.pl
4184 raw.pl
3626 screen.pl
4212 subcheck.pl
4251 util.pl
4196 vars.pl
The toolsver utility is new with CPU2006.
You can invoke toolsver as:
specperl %spec%\bin\toolsver rawfile
Copyright (C) 1999-2006 Standard Performance Evaluation Corporation
All Rights Reserved