bug#23521: XFAIL

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

bug#23521: XFAIL

Reuben Thomas
The documentation says: "It's not uncommon, especially during early development stages, that some tests fail for known reasons, and that the developer doesn't want
to tackle these failures immediately (this is especially true when the
failing tests deal with corner cases)."

Another common use for "expected failure" is to write tests to check that error conditions arise as expected, for example, by checking that a program raises an error when given invalid input.

If that's a reasonable use of automake's test harness, perhaps the documentation could reflect that, e.g. by adding:

"Another use for XFAIL is to mark tests that are supposed to fail, for example, to check that a program raises an error when given invalid input."

It is often easier to write expected-to-fail tests this way (so that they can all look the same), rather than have to have, for example, an extra driver that converts expected errors into success codes for the automake test harness.

--
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

bug#23521: XFAIL

Mathieu Lirzin
Hi,

Reuben Thomas <[hidden email]> writes:

> The documentation says: "It's not uncommon, especially during early
> development stages, that some tests fail for known reasons, and that
> the developer doesn't want
> to tackle these failures immediately (this is especially true when the
> failing tests deal with corner cases)."
>
> Another common use for "expected failure" is to write tests to check
> that error conditions arise as expected, for example, by checking that
> a program raises an error when given invalid input.

I agree that XFAIL can be ambiguous, however I think this usage is not
desirable.  It gives an additional opposite meaning to XFAIL symbol
which makes it even more confusing.

> If that's a reasonable use of automake's test harness, perhaps the
> documentation could reflect that, e.g. by adding:
>
> "Another use for XFAIL is to mark tests that are supposed to fail, for
> example, to check that a program raises an error when given invalid
> input."
>
> It is often easier to write expected-to-fail tests this way (so that
> they can all look the same), rather than have to have, for example, an
> extra driver that converts expected errors into success codes for the
> automake test harness.

What do you mean precisely by “an extra driver”?

Thanks

--
Mathieu Lirzin



Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

bug#23521: XFAIL

Peter Johansson-6


On 05/19/2016 09:04 AM, Mathieu Lirzin wrote:
>> Another common use for "expected failure" is to write tests to check
>> >that error conditions arise as expected, for example, by checking that
>> >a program raises an error when given invalid input.
> I agree that XFAIL can be ambiguous, however I think this usage is not
> desirable.  It gives an additional opposite meaning to XFAIL symbol
> which makes it even more confusing.
>
I agree. When I wanna tests that a program fails with incorrect input, I
prefer writing a tests that calls the program, check that it fails (exit
1 or whatever is expected), and perhaps even parse the error message,
and if it looks like I expect exit 0 aka PASS.

Cheers,
Peter



Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

bug#23521: XFAIL

Reuben Thomas
In reply to this post by Mathieu Lirzin
On 19 May 2016 at 00:04, Mathieu Lirzin <[hidden email]> wrote:
> It is often easier to write expected-to-fail tests this way (so that
> they can all look the same), rather than have to have, for example, an
> extra driver that converts expected errors into success codes for the
> automake test harness.

What do you mean precisely by “an extra driver”?

​A custom test driver.​

--
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

bug#23521: XFAIL

Reuben Thomas
In reply to this post by Peter Johansson-6
On 19 May 2016 at 00:55, Peter Johansson <[hidden email]> wrote:


On 05/19/2016 09:04 AM, Mathieu Lirzin wrote:
Another common use for "expected failure" is to write tests to check
>that error conditions arise as expected, for example, by checking that
>a program raises an error when given invalid input.
I agree that XFAIL can be ambiguous, however I think this usage is not
desirable.  It gives an additional opposite meaning to XFAIL symbol
which makes it even more confusing.

I agree. When I wanna tests that a program fails with incorrect input, I prefer writing a tests that calls the program, check that it fails (exit 1 or whatever is expected), and perhaps even parse the error message, and if it looks like I expect exit 0 aka PASS.

​Thanks. I shall continue with my "deviant" usage for now, but if that is not considered normal, I understand that you won't want to document it.

--
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

bug#23521: XFAIL

Gavin Smith
In reply to this post by Mathieu Lirzin
On 19 May 2016 at 00:04, Mathieu Lirzin <[hidden email]> wrote:
>> It is often easier to write expected-to-fail tests this way (so that
>> they can all look the same), rather than have to have, for example, an
>> extra driver that converts expected errors into success codes for the
>> automake test harness.
>
> What do you mean precisely by “an extra driver”?

This would be a reference to a "custom test driver".
https://www.gnu.org/software/automake/manual/html_node/Overview-of-Custom-Test-Drivers-Support.html#Overview-of-Custom-Test-Drivers-Support



Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

bug#23521: XFAIL

Reuben Thomas
On 20 May 2016 at 15:58, Gavin Smith <[hidden email]> wrote:
On 19 May 2016 at 00:04, Mathieu Lirzin <[hidden email]> wrote:
>> It is often easier to write expected-to-fail tests this way (so that
>> they can all look the same), rather than have to have, for example, an
>> extra driver that converts expected errors into success codes for the
>> automake test harness.
>
> What do you mean precisely by “an extra driver”?

This would be a reference to a "custom test driver".
https://www.gnu.org/software/automake/manual/html_node/Overview-of-Custom-Test-Drivers-Support.html#Overview-of-Custom-Test-Drivers-Support

​Thanks, that's what I meant.

--
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

bug#23521: XFAIL

Mathieu Lirzin
In reply to this post by Reuben Thomas
Reuben Thomas <[hidden email]> writes:

> On 19 May 2016 at 00:04, Mathieu Lirzin <[hidden email]> wrote:
>
>     > It is often easier to write expected-to-fail tests this way (so
>     that
>     > they can all look the same), rather than have to have, for
>     example, an
>     > extra driver that converts expected errors into success codes
>     for the
>     > automake test harness.
>    
>     What do you mean precisely by “an extra driver”?
>    
>
> ​A custom test driver.​

OK, I wasn't sure.  Indeed a custom test driver seems a bit heavy just
checking failures.  IMO the solution Peter proposed is nice and simple.

--
Mathieu Lirzin



Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

bug#23521: XFAIL

Reuben Thomas
On 20 May 2016 at 16:49, Mathieu Lirzin <[hidden email]> wrote:
Reuben Thomas <[hidden email]> writes:

> On 19 May 2016 at 00:04, Mathieu Lirzin <[hidden email]> wrote:
>
>     > It is often easier to write expected-to-fail tests this way (so
>     that
>     > they can all look the same), rather than have to have, for
>     example, an
>     > extra driver that converts expected errors into success codes
>     for the
>     > automake test harness.
>
>     What do you mean precisely by “an extra driver”?
>
>
> ​A custom test driver.​

OK, I wasn't sure.  Indeed a custom test driver seems a bit heavy just
checking failures.  IMO the solution Peter proposed is nice and simple.

​What Peter proposed is essentially a custom test driver: I would not expect to duplicate the logic to check the return code &c. in each test expected to fail; rather, I would put it in a custom test driver that would handle expected fails and mark them as passes. (My expected fails are all of the same type, i.e. a non-zero exit code. It might additionally be useful, as Peter suggests, to check that an expected error message is produced.)

--
Loading...