A small set of test files
When I started out looking for test files to use to compare accessibility testing tools, I was (of course) attracted to the ATR tests, developed by Chris Redpath at the Technology Resource Center at the University Of Toronto (ATRC, http://atrc.utoronto.ca/ ). Those tests consist of about 220 files ( http://checker.atrc.utoronto.ca/ ) including at least one file and often several files for each of the WCAG Checkpoints, Version 1.0. Although still in working draft form, there are also tests for Version 2.0 of the Web Content Accessibility Guidelines.
My main concerns with the ATR tests were that there were so many files and that many of the test files are nearly empty.
For example some files relating to <object> elements had merely <object /> in the
code; I think a checker could—maybe even should—recognize that this is an empty object and raise no warning or error.
So I decided to create a much smaller and more manageable set of files (covering many fewer conditions) in which each file contains more content and generally there are pairs of files to compare the tool response for the error condition and the error condition corrected. Sometimes the ATR files are paired, one file with the error condition, the other without the error condition—not with the error corrected.
The test suite
The test suite is arranged by issues, not by files. So there is the “Alt-text”
issue (#1 below) and there are two files associated with it. One in which the image has no alt-text (step-1-1-1-f.htm )
and essentially the same file with the alt attribute added (step-1-1-1-s.htm ) with respectable
alt-text. The file naming system stems from an earlier version of the ATR tests. The file name includes WCAG checkpoint
number, test number for that checkpoint, and an
f for failure or s for success.
A listing of all the files is at https://jimthatcher.com/step/new. All the files referred to in the discussion below are available in that directory too, for example, https://jimthatcher.com/step/new/step-1-1-1-f.htm, which will be referred to just as step-1-1-1-f.htm.
1. Alt-text.
Every image element, img, must have an alt attribute. No alt-text (step-1-1-1-f.htm)
is an error and there should be no error when alt-text is present (step-1-1-1-s.htm).
Results: All tools passed this test. There is no surprise here.
2. ASCII art for alt-text.
If alt-text is ASCII art, like alt="-->", a screen reader would speak hyphen
hyphen greater than which is not meaningful. It is an error when alt-text consists of ASCII art (step-1-1-1-f2).
Results: On my first appraisal of the results of this test I thought that InFocus caught
the error. I was concerned because the error was Alt text required which means that the alt attribute
is missing. When I tried different ASCII art, namely alt=":-)", InFocus did not pick it up as an
error. From the highlighting of the source code in the original case (alt="-->" ) it turned
out that InFocus was prematurely terminating the <img> tag with the ">" at
the end of the value of the alt attribute – and in fact it did not catch this error.
For each of its alt-text errors, LIFT has both, “image should have valid alt” and “image should have equivalent alt.” Lacking valid alt is an error while the equivalent alt message is a human review warning saying it is necessary to compare the image with the alt-text. Since that warning is raised on any significant image, it would seem that LIFT would have to say that the ASCII art for alt-text is not valid. LIFT did not do that. Compare with #5 where LIFT declares alt-text longer than 150 characters to be “not valid”.
WebKing and Ramp were the only tools to pass this test, both changes in the tools from the original tests.
3. Object requires default content.
An object element should have content which is available if the object/plug-in is not supported (step-1-1-2-s.htm).
Though this is a requirement of WCAG 1.0, it is not widely supported by assistive technology. A testing tool scores on this
issue if there is no warning or error on the good file (step-1-1-2-s.htm)
and there is a warning or error on the bad one (step-1-1-2-f.htm)
which has no default content.
Results: All tools passed this test.
4. Image Button.
Every image button, that is input element with type="image", must have alt-text that
specifies the purpose of the button (step-1-1-4-s, step-1-1-4-f).
Results: Technically all six tools caught this error, but WebXM received a failing grade
because of the way it presented the issue. WebXM raised the form labeling error for this test: Explicitly
associate form controls and their labels with the <label> element. That error is not raised for the file which has alt-text on the image button (step-1-1-4-s).
In this version, WebXM considers the alt-text on an image button to be a form labeling problem, not an alt-text problem.
Both test files have an input element with type="text" which is
correctly labeled. InFocus raises the error Label should be placed next to input whenever
there is an input element; even when the prompt is perfectly placed and labeled. We know that this stems
from Web Content Accessibility 1.0 Checkpoint 10.2:
Until user agents support explicit associations between labels and form controls, for all form controls with implicitly associated labels, ensure that the label is properly positioned.
But a testing tool should be able to estimate positioning and raise this issue only when it is really necessary, especially because user agents do support explicit associations between labels and form controls.
5. Long alt-text.
Alt-text should be short, succinct, and to the point. The question is “how long is too long?” I have chosen 151 characters
(step-1-1-5-f) as the break-off point that should raise an error. There is also a file with 102 characters of
alt text (step-1-1-5-f2) which I think should be an error too.
Results: All of the tools caught this error – alt-text longer than 150 characters. None of the tools raised an error for 102 characters. Several tools provide user specification of the maximum length (InFocus, Ramp, and LIFT) along with other characteristics of acceptable alt-text.
InFocus raises the violation, Valid alt attribute required . As mentioned above, LIFT also raises an error about validity of the alt-text. What is troubling, of course, is that the alt-text at 151 characters is certainly valid, but too long. The detailed explanation of the InFocus violation softens the concern with This image has an invalid or suspicious alt attribute . On the other hand, Bobby warns, Alt text > 150 characters, consider providing a separate description, making the statement of the error/warning perfectly clear.
6. Image map area.
Every area element of a map for a client-side image map needs to have alt-text. The sample file
(step-1-1-6-f) has one <area> tag that has no alt-text. For comparison, step-1-1-6-s has
alt-text on all the area elements.
Results: All tools got this right. WebXM does fine on this test but I noticed that the
test itself, "Provide alternative text for all image map hot-spots," is also listed in the section of the report
called Passed Evaluations. The reason the test appears in the list of evaluations that
passed is that there are several image map area elements that do have alt-text and for those elements the
evaluation passed. I think that the idea should be that the page passes the “image map hot spot” test if all image map
hot spots have appropriate alt-text and fails if any one does not have alt-text.
7. Server-side image maps.
I believe that a server side map (<img> with ismap attribute) should never be used, so
any page with a server-side map should raise an error. There are two test files, one with the server-side map and no text
links (step-1-2-f) and, just for interest, step-1-2-s is coded with text links provided for each server side
hot spot. The tools get this right if they raise an error or warning for the server-side map (on either or both files).
Results: All tools got this right but usually by raising a warning. Bobby has a warning, Provide text links that duplicate all server-side image map hot-spots. It is a warning because the tool can’t tell whether or not the duplicate links are available. I think that this should be an error or at least a stronger statement about the problem with server-side image maps. Checkpoint 9.1 of WCAG and §1194.22(e) of Section 508 use the same wording concerning the use of server-side maps:
Provide client-side image maps instead of server-side image maps except where the regions cannot be defined with an available geometric shape.
It may be impractical to use client-side image maps but all regions can be defined by available geometric shapes. Ramp’s response is also a warning, but the wording is better: Use client-side image maps instead of server side maps .
8. Frame titles.
Each <frame> in a <frameset> needs a title attribute indicating the purpose of the
frame. In the sample file (step-12-1-f) the <frame> elements do not have title attributes.
For comparison, step-12-1-s has adequate title attributes. (<iframe> is treated
separately in test #40.)
Results: All tools got this right.
9. Quality of frame titles.
The title attributes on the <frame> elements must be meaningful. Here (step-12-1-f2)
the sample frames have title="top", title="left" and title="right" which
are not meaningful—that is they do not specify the purpose of the frame.
Results: On the first round of tests only InFocus go this right. With updates, Ramp
and WebKing raise errors here too. Again, like test #5, some tools allow personalizing the words or phrases that will not be
allowed in the title attributes of the frame elements.
10. Input Element needs label.
The first form example (step-12-4-f.htm) is just a text input field (type="text")
with neither a title attribute nor an associated label element. This condition should raise an
error. The same form using the label element (step-12-4-s.htm) is correct and should not raise the error.
Results: All tools got this right.
11. Use of title attribute for form control.
Another way to explicitly specify prompting text for a text input field is to use the title attribute
on the input control. This is illustrated in step-12-4-s4.htm. There should be no error or warning for this file.
Results: Only three out of six tools got this right, InFocus, Ramp and WebKing. This
is disappointing. As we argued in Chapter 8 there are situations where the title attribute is the only reasonable
way to precisely label a form control.
12. Text intervenes between label and control.
The text that is enclosed by the label element does not have to be right next to the control. Here
the label element encloses text, but there is intervening text before the control is encountered. This file
(step-12-4-s3.htm) should not raise an error or warning.
Results: InFocus insists that prompting text must be properly placed. Using the label element
frees the developer from that requirement by programmatically connecting the prompt with the control at least for people
using assistive technology. For others with cognitive impairments or low vision the positioning can be critical. Unfortunately,
InFocus raises this warning even if the prompt is perfectly placed. Since we are checking to see if the tool accepts a
label coming from a place that is not next to the control, we conclude that InFocus failed this test. The 5 other
tools passed.
13. Prompting text from two places.
Not only does the labeling text not have to be adjacent; it can come from two places. It is perfectly legitimate to have
two label elements with the same for attribute which this file (step-12-4-s6) illustrates.
As we have seen in Chapter 8, AT support for two labels is spotty at best, but this should not be an error.
Results: All tools got this right. I suspect that the tools don’t even notice that there are two labels. Since support by screen readers for this situation is so poor, this probably should be deemed an error.
14. An invisible gif holds the prompting text.
A common phenomenon on http://www.ibm.com is to use an invisible gif with alt-text as
the prompt for the search form text entry field. This technique was introduced by IBM at a time when few tools recognized
the title attribute (see #11 above). Although this technique is acceptable—it is not an error—the title attribute
is much better (see Chapter 8). The file, step-12-4-s5, uses an invisible gif with alt-text in
the label container. It should not raise an error.
Results: All tools got this right but there were two minor problems. InFocus continues
to warn that prompting text must be properly placed and WebKing raises the error that a spacer image requires alt="".
When it is clear that the “spacer image” is conveying information as is the case here, then that warning should be turned
off.
15. Label that matches no control.
This form example, step-12-4-f2, has a label element, but the for attribute does
not match any id. This is an error.
Results: All tools got this right.
16. Two input elements with same id
In this case (step-12-4-f3) there are two input elements with the same id. This
is an error. Not only is it an accessibility error; using the same id on any two elements is invalid HTML.
Results: Surprisingly this was detected only by LIFT on the first pass over the tests. The newer versions of WebKing and Ramp now find this error. Three tools got this right.
17. Textarea needs label or title.
This file, step-12-4-f8, contains a textarea with no label element or title attribute.
This test is just like #10 except for textarea instead of input with type="text" .
Results: All tools got this right.
18. Select menu needs label or title.
This file, step-12-4-f9, contains a select menu with no label element or title attribute.
Just like #10 except for select instead of input with type="text".
Results: All tools got this right.
19. Inaccessible select menu.
This file, step-12-4-f10, contains a select menu which is open (size="4")
and which has an onChange event handler which opens a new page. This menu is not accessible because it cannot
be operated with the keyboard. As soon as you press the Down Arrow when the menu has focus, it fires and the new page is
opened. This is an error, but the tool scores here only if it does not flag step-12-4-s10 as an error. That
file is the same except the select menu is not “open” (size="1") which can be handled
from the keyboard using Alt +Down Arrow then Up and Down arrows.
Results: Ramp raises an error on both files. Only WebKing got it right.
20. Empty label.
This file (step-12-4-f11) has a properly coded label element except that the contents of that label element
are empty. This is an error.
Results: It is surprising that apparently only two tools (WebKing and WebXM) were looking at all at the “quality” of the label on the first pass through the tests. Version 6.0 of Ramp now looks at that aspect of form labeling so now three tools get it right.
21. Inadequate link text.
Link text is “click here” (step-13-1-1-f ); an error.
Results: Two tools missed this, InFocus and LIFT. Bobby flags an error for the “Click here” link, Create link phrases that make sense when read out of context . The text as a warning (see #22) appears any time links are present.
22. Image link with empty alt-text.
An extreme case of inadequate link text, an anchor containing only one image and that image has empty alt text, alt="" (step-13-1-1-f2 ).
Results: InFocus is the only tool to fail this test (and #23). Ramp lists the Violation as invalid text equivalent for image . It would seem that if the text equivalent were invalid, it would be invalid in #24, which it is not. I think this error should be invalid (empty) link text.
WebXM raises the error Create link phrases that make sense when read out of
context for the situation of an image link in which the alt-text is empty and also for the case (#23) when the
alt-text is just spaces. The error description is directly from the Web Content Accessibility Guidelines, but these situations
seem to me to be worse than the inability to read out of context. There is no link text! On the other hand when there
is no link text JAWS reads the image source file name, which in this case is Resources, giving good information
about the link. One certainly should not count on that.
Bobby raises the error, Provide alternative text for all images , for this file, and also the warning (not error as in #21), Make sure that all link phrases make sense when read out of context . This warning is raised for all files that have links. And the image has alt text, it is not adequate.
23. Image link with spaces for alt-text.
Very similar to number #22, this anchor element (<a>) contains just one img element
and the alt-text on that is alt=" " (quote four-spaces quote), step-13-1-1-f3.
Results: InFocus failed this test, but WebKing got it right even though it failed #22 (image link with empty alt-text).
24. Link with empty alt-text but also text.
This file, step-13-1-1-s2, is OK. In this commonly occurring situation, there is in image in the anchor with alt="" but
there is also text in the anchor. This file should pass, but a tool gets a score here only if it passes this file and fails
#22. The issue here is distinguishing between empty link text and non-empty link text.
Results: Two tools failed this test, Bobby and InFocus. Bobby raises the same errors for this file as for the one with empty alt text and no text in the anchor, and so fails this test.
25. Link text; “click here” with title attribute.
This file (step-13-1-1-s3 ) has an anchor element (<a> ) containing exactly the text click
here which is inadequate (see #21). But there is also a title attribute on the anchor which specifies
the target of the link. This is not an error. A tool scores here only if this file passes and #21 fails.
Results: Two tools got this right, Ramp and WebKing. Watchfire informed me that WebXM has an option to look for the title attribute on links but that is switched off in the default settings which I used.
26. Same link text, different targets.
If two links have the same link text but different targets (href) then the links cannot be distinguished
in a list of links. This file, step-13-1-2-f, has two links with the same text but different href.
It is an error.
Results: All tools got this right.
27. Page title.
The content of the title element of a page is what appears in the title bar of the browser and it is spoken
by a screen reader when the page opens. Every page should have a non-empty title element. This file, step-13-2-f.htm,
has no title element which is an error.
Results: InFocus and WebKing were the only tools that did not raise an error in this case.
28. Inadequate page title.
This test is similar to #27, but here the title is Title, i.e., the code looks like this: <title>Title</title>. The file, step-13-2-f2, should
raise an error.
Results: Only Ramp got this right the first go round; WebKing joined the circle of winners with their newer version. Four tools failed this test.
29. Skip link.
The two files both have links at the top of the page followed by text. The file that fails has neither headings nor a skip
link after the “navigation links” (step-15-2-f). The file that is OK has a skip link (step-15-2-s2).
A tool scores on this test if it raises an appropriate error on the file with no skip link, step-15-2-f, and
does not raise that error on step-15-2-s2 which has the skip link.
Results: Three tools got this right, InFocus, LIFT and Ramp.
30. Heading for skipping to content.
This is exactly like #29 except the successful file (step-15-2-s.htm)
has headings, especially a level 1 heading at the beginning of the main content.
Results: I was disappointed that, on the first pass through the tests, none of the tools that were looking for skip techniques at all recognized headings as such a technique. Now Ramp Ascend 6.0 sees a heading after a block of links as a technique for skipping those links. This test can be turned on and off in the Accessibility Options of Ramp. I hope more tools will recognize this option because headings markup provides a much better technique for addressing page navigation than skip links (see Chapter 7).
31. Layout table that won’t resize.
When a layout table uses pixel widths it won’t resize with either a larger window or larger text. This file, step-3-4-f, uses
fixed widths in pixels. The response by the tool needs to be compared with that for a file which uses percentage widths instead
of pixel widths (step-3-4-s). The tool is successful if it raises an error for the table with fixed widths
and no error on the table with percentage widths.
Results: All tools got this right, though Ramp failed on the first pass through the tests.
32. Data tables.
The issue I wanted to check was whether the tools checked for data tables - distinguishing then from layout tables. The data
table
(step-5-1-s) is quite obviously such and marked up properly. A tool scores here if there is no data table
error or warning on the good table or on the layout table (step-5-1-s3 ) and a table markup warning or error
on the one with no use of TH (step-5-1-f).
Results: Four tools got this right; Bobby and WebXM failed. Ramp raises an interactive check for the user to confirm that a table is a data table. But Ramp is successful because it is raising that test when it suspects a table is a data table, and not on all tables. So on our examples it raised the interactive check on the data tables and not on the layout tables.
33. Layout table with a summary.
It is an error for a layout table (step-5-1-f3) to have a summary attribute and/or th markup.
Results: Only Bobby and Ramp got this right. Bobby raises a warning, If
this table is used for layout only, do not use structural markup to achieve formatting effects, and it does not
raise that warning when a layout table does not have structural markup. Ramp raises two errors, one for the summary attribute
(Tables used for layout should not have summary) and one for the table head (th)
markup (Tables used for layout should not use headings to create formatting effects).
34. Frame source must be HTML.
Here (step-6-2-f ) is a frame page (<frameset>) in which the first frame directly
references an image (.gif ). The src attributes of frame must be an HTML page.
Results: All tools except LIFT got this right. The wording of the InFocus error in this case was awkward: use markup file as source . But the tool definitely caught the problem.
35. Blink element.
Here is a file, step-7-2-f, that uses the blink element. This is an error because the result is
blinking text; also because the blink element is deprecated.
Results: All tools got this right.
36. Marquee element.
Here’s a file, step-7-3-f, that uses the marquee element. Like #35, this is an error because
text is moving and because the marquee element is deprecated.
Results: All tools got this right.
37. Auto refresh.
This page (step-7-4-f ) has an auto-refresh every 6 seconds, an error.
Results: All tools except InFocus and WebXM raised an error for the auto-refresh.
38. Keyboard access.
The issue with this file, step-9-3-f, is keyboard access. It contains a JavaScript function that requires a
double click (onDoubleclick) and it doesn’t work from the keyboard. A tool scored here if it raises a scripting
error on this file (no keyboard access) but not on step-9-3-s where onKeyDown is used for the
same function.
Results: Only Bobby got this right.
39. Structure of headings markup.
WCAG suggests that headings should be used according to specification. There are two files for testing the use of headings.
The correct one, step-3-5-s, has these headings (in order) h1, h2, h2, h3, h1 .
The error file, step-3-5-f, has h4, h1, h2, h5, h1.
There are two errors here, the transition h4 (starting) to h1, and h2 followed by h5 .
The use of headings is so important for page navigation, errors like these are not important in my opinion.
Results: LIFT and WebKing failed this test. Deque pointed out that AERT techniques ( http://www.w3.org/TR/2000/WD-AERT-20000426 ) specify that the requirement on header nesting
to be that "Header levels must not increase by more than 1 level." So there is only one error in the subject file, step-3-5-f.htm, which
is the transition form h2 to h5, by this specification.
For pages with no headings markup whatsoever, LIFT warns,
Use header elements according to specification. Yet neither of the pages with headings
markup (step3-5-s and step3-5-f) get that warning. It seems totally backwards to me.
40. Title for in-line frame.
This is another frame issue, an iframe without a title attribute (step-12-1-f3) .
Results: All the tools got this right except InFocus.