1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
|
# c-testsuite
This project aims to be a collaborative database of C compiler test cases,
minimal test runners, and public test results. The general idea is that
the various test suites here adhere to well defined, simple interfaces and projects can contribute
tests back that match those interfaces, or use those tests by implementing runner scripts matching
the specifications.
There are many tools that may benefit such as C compilers, transpilers, interpreters and emulators, so we should seek to agree on simple easy test interfaces, and implement
those interfaces for a variety of tools.
results are published daily to https://c-testsuite.github.io/
# Test suites
The top level test-suite runners output https://testanything.org output.
## single-exec suite
entry point is ```./single-exec```
### runners/single-exec/*
The runner will be invoked as:
```
$ ./runners/single-exec/$NAME test/single-exec/case.c
```
The runner is free to output any data it wants, but must return
nonzero on failure.
The runner will be considered a failure if it takes more than 5 minutes.
### tests/single-exec/*
- Single .c file tests.
- 'main' is the entry point.
- The file $t.c.expected must match stdout+stderr of
the test.
- The test programs exit with 0 on success.
C standard, Portability, preprocessor and libc requirements
are specified via tags that can be filtered against using
search queries, that can generate skip lists.
### Example:
```$ ./single-exec gcc-x86_64 ```
# Skipping tests
Try to skip a test if your compiler platform can NEVER pass it, the test is not appropriate.
In that case, there is only one mechanism, add a command that prints a list of tests to skip
on stdout named:
```
./runners/*/$TOOL.skip
```
# Search and query
All tests have a matching $t.tags file. This file specifies attributes
of the test that can be filtered and queried.
The query language is based off of https://github.com/oniony/TMSU tags.
The query language grammar is shown here:
https://github.com/oniony/TMSU/blob/master/misc/ebnf/query.ebnf
Support tags are currently
```
suite={single-exec, ...}
portable
The test should be portable C.
arch-x86_64
The test should pass on x86_64
c89
c99
c11
needs-cpp
Test relies on the preprocessor
needs-libc
Test relies on libc
```
Implicit tags:
c89 implies c99 and c11
c99 implies c11
example query:
```
$ ./scripts/make-search-index
$ ./scripts/search-tests "c99 suite=single-exec (portable or arch-amd64)"
```
These queries can be used to generate skip lists.
# otags files
otag files are intended to allow
a tests origin to be discovered. They contain the following fields.
```
org=$DOMAINNAME
repository=$SRCURL
version=$UNIQUE_VERSION
path=$TEST_PATH_IN_REPOSITORY
```
# Dependencies
Running tests
- posix sh
- python3
- coreutils
- tool under test
Querying tests:
Currently test search is based on
https://github.com/oniony/TMSU
We are sympathetic to those who do not wish to deal with
installing a lesser known third party tool, so will think of
ways to ease the burden in the future.
# Tips
## Naming test cases
The names are not stable for now, so if you
refer to a test case in your issue tracker, it is best to
name it something like ```c-testsuite/$CTESTGITCOMMIT/path/to/test```.
## Getting a summary from the command line
./single-exec $runner | ./scripts/tapsummary | head
## The full TAP test suites report can be viewed or 'curl'ed
For example:
- https://c-testsuite.github.io/gcc-x86_64_report.html
- https://c-testsuite.github.io/gcc-x86_64-single-exec_report.tap
- https://c-testsuite.github.io/gcc-x86_64-single-exec_report.tap.txt
|
