blob: 7a96c25c72ad8ee864c4b0cdd657ae1652b1ae10
1 | \input texinfo @c -*- texinfo -*- |
2 | @documentencoding UTF-8 |
3 | |
4 | @settitle FFmpeg Automated Testing Environment |
5 | @titlepage |
6 | @center @titlefont{FFmpeg Automated Testing Environment} |
7 | @end titlepage |
8 | |
9 | @node Top |
10 | @top |
11 | |
12 | @contents |
13 | |
14 | @chapter Introduction |
15 | |
16 | FATE is an extended regression suite on the client-side and a means |
17 | for results aggregation and presentation on the server-side. |
18 | |
19 | The first part of this document explains how you can use FATE from |
20 | your FFmpeg source directory to test your ffmpeg binary. The second |
21 | part describes how you can run FATE to submit the results to FFmpeg's |
22 | FATE server. |
23 | |
24 | In any way you can have a look at the publicly viewable FATE results |
25 | by visiting this website: |
26 | |
27 | @url{http://fate.ffmpeg.org/} |
28 | |
29 | This is especially recommended for all people contributing source |
30 | code to FFmpeg, as it can be seen if some test on some platform broke |
31 | with their recent contribution. This usually happens on the platforms |
32 | the developers could not test on. |
33 | |
34 | The second part of this document describes how you can run FATE to |
35 | submit your results to FFmpeg's FATE server. If you want to submit your |
36 | results be sure to check that your combination of CPU, OS and compiler |
37 | is not already listed on the above mentioned website. |
38 | |
39 | In the third part you can find a comprehensive listing of FATE makefile |
40 | targets and variables. |
41 | |
42 | |
43 | @chapter Using FATE from your FFmpeg source directory |
44 | |
45 | If you want to run FATE on your machine you need to have the samples |
46 | in place. You can get the samples via the build target fate-rsync. |
47 | Use this command from the top-level source directory: |
48 | |
49 | @example |
50 | make fate-rsync SAMPLES=fate-suite/ |
51 | make fate SAMPLES=fate-suite/ |
52 | @end example |
53 | |
54 | The above commands set the samples location by passing a makefile |
55 | variable via command line. It is also possible to set the samples |
56 | location at source configuration time by invoking configure with |
57 | @option{--samples=<path to the samples directory>}. Afterwards you can |
58 | invoke the makefile targets without setting the @var{SAMPLES} makefile |
59 | variable. This is illustrated by the following commands: |
60 | |
61 | @example |
62 | ./configure --samples=fate-suite/ |
63 | make fate-rsync |
64 | make fate |
65 | @end example |
66 | |
67 | Yet another way to tell FATE about the location of the sample |
68 | directory is by making sure the environment variable FATE_SAMPLES |
69 | contains the path to your samples directory. This can be achieved |
70 | by e.g. putting that variable in your shell profile or by setting |
71 | it in your interactive session. |
72 | |
73 | @example |
74 | FATE_SAMPLES=fate-suite/ make fate |
75 | @end example |
76 | |
77 | @float NOTE |
78 | Do not put a '~' character in the samples path to indicate a home |
79 | directory. Because of shell nuances, this will cause FATE to fail. |
80 | @end float |
81 | |
82 | To use a custom wrapper to run the test, pass @option{--target-exec} to |
83 | @command{configure} or set the @var{TARGET_EXEC} Make variable. |
84 | |
85 | |
86 | @chapter Submitting the results to the FFmpeg result aggregation server |
87 | |
88 | To submit your results to the server you should run fate through the |
89 | shell script @file{tests/fate.sh} from the FFmpeg sources. This script needs |
90 | to be invoked with a configuration file as its first argument. |
91 | |
92 | @example |
93 | tests/fate.sh /path/to/fate_config |
94 | @end example |
95 | |
96 | A configuration file template with comments describing the individual |
97 | configuration variables can be found at @file{doc/fate_config.sh.template}. |
98 | |
99 | @ifhtml |
100 | The mentioned configuration template is also available here: |
101 | @verbatiminclude fate_config.sh.template |
102 | @end ifhtml |
103 | |
104 | Create a configuration that suits your needs, based on the configuration |
105 | template. The @env{slot} configuration variable can be any string that is not |
106 | yet used, but it is suggested that you name it adhering to the following |
107 | pattern @samp{@var{arch}-@var{os}-@var{compiler}-@var{compiler version}}. The |
108 | configuration file itself will be sourced in a shell script, therefore all |
109 | shell features may be used. This enables you to setup the environment as you |
110 | need it for your build. |
111 | |
112 | For your first test runs the @env{fate_recv} variable should be empty or |
113 | commented out. This will run everything as normal except that it will omit |
114 | the submission of the results to the server. The following files should be |
115 | present in $workdir as specified in the configuration file: |
116 | |
117 | @itemize |
118 | @item configure.log |
119 | @item compile.log |
120 | @item test.log |
121 | @item report |
122 | @item version |
123 | @end itemize |
124 | |
125 | When you have everything working properly you can create an SSH key pair |
126 | and send the public key to the FATE server administrator who can be contacted |
127 | at the email address @email{fate-admin@@ffmpeg.org}. |
128 | |
129 | Configure your SSH client to use public key authentication with that key |
130 | when connecting to the FATE server. Also do not forget to check the identity |
131 | of the server and to accept its host key. This can usually be achieved by |
132 | running your SSH client manually and killing it after you accepted the key. |
133 | The FATE server's fingerprint is: |
134 | |
135 | @table @samp |
136 | @item RSA |
137 | d3:f1:83:97:a4:75:2b:a6:fb:d6:e8:aa:81:93:97:51 |
138 | @item ECDSA |
139 | 76:9f:68:32:04:1e:d5:d4:ec:47:3f:dc:fc:18:17:86 |
140 | @end table |
141 | |
142 | If you have problems connecting to the FATE server, it may help to try out |
143 | the @command{ssh} command with one or more @option{-v} options. You should |
144 | get detailed output concerning your SSH configuration and the authentication |
145 | process. |
146 | |
147 | The only thing left is to automate the execution of the fate.sh script and |
148 | the synchronisation of the samples directory. |
149 | |
150 | |
151 | @chapter FATE makefile targets and variables |
152 | |
153 | @section Makefile targets |
154 | |
155 | @table @option |
156 | @item fate-rsync |
157 | Download/synchronize sample files to the configured samples directory. |
158 | |
159 | @item fate-list |
160 | Will list all fate/regression test targets. |
161 | |
162 | @item fate |
163 | Run the FATE test suite (requires the fate-suite dataset). |
164 | @end table |
165 | |
166 | @section Makefile variables |
167 | |
168 | @table @env |
169 | @item V |
170 | Verbosity level, can be set to 0, 1 or 2. |
171 | @itemize |
172 | @item 0: show just the test arguments |
173 | @item 1: show just the command used in the test |
174 | @item 2: show everything |
175 | @end itemize |
176 | |
177 | @item SAMPLES |
178 | Specify or override the path to the FATE samples at make time, it has a |
179 | meaning only while running the regression tests. |
180 | |
181 | @item THREADS |
182 | Specify how many threads to use while running regression tests, it is |
183 | quite useful to detect thread-related regressions. |
184 | |
185 | @item THREAD_TYPE |
186 | Specify which threading strategy test, either @samp{slice} or @samp{frame}, |
187 | by default @samp{slice+frame} |
188 | |
189 | @item CPUFLAGS |
190 | Specify CPU flags. |
191 | |
192 | @item TARGET_EXEC |
193 | Specify or override the wrapper used to run the tests. |
194 | The @env{TARGET_EXEC} option provides a way to run FATE wrapped in |
195 | @command{valgrind}, @command{qemu-user} or @command{wine} or on remote targets |
196 | through @command{ssh}. |
197 | |
198 | @item GEN |
199 | Set to @samp{1} to generate the missing or mismatched references. |
200 | |
201 | @item HWACCEL |
202 | Specify which hardware acceleration to use while running regression tests, |
203 | by default @samp{none} is used. |
204 | |
205 | @end table |
206 | |
207 | @section Examples |
208 | |
209 | @example |
210 | make V=1 SAMPLES=/var/fate/samples THREADS=2 CPUFLAGS=mmx fate |
211 | @end example |
212 |