How to avoid stopping the recursive `check` target?

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

How to avoid stopping the recursive `check` target?

Philippe Proulx
My project has the following directory structure (some files hidden for
clarity):

    Makefile.am
    tests/
      Makefile.am
      a/
        Makefile.am
      b/
        Makefile.am
      c/
        Makefile.am

If I put all the tests to run that are in the a/, b/, and c/
subdirectories in the TESTS variable of tests/Makefile.am, then with
`make check` in tests/, all the tests run, even if one test in one of
the subdirectories fail. Assume this is the behaviour I want.

With this configuration, however, I cannot go to b/, for example, and
run `make check`: since there's no TESTS variable there, `make check`
does nothing.

So I put a TESTS variable in a/Makefile.am, b/Makefile.am, and
c/Makefile.am, and remove the TESTS variable from tests/Makefile.am. Now
I can go to a/, b/, or c/ and run `make check` to test only specific
parts of the project. However, since each individual `make check` can
fail, now the "global" `make check` in tests/ fails as soon as one
subdirectory fails, which is the expected behaviour of Make.

My current workaround is to use `make --keep-going check` in tests/, so
that, as per make(1):

    Continue as much as possible after an error.  While the target that
    failed, and those that depend on it, can‐ not be remade, the other
    dependencies of these targets can be processed all the same.

This seems to run all the individual `make check` and exit with
something else than 0 if one of them fails.

I can also wrap this `make --keep-going check` in a new target, for
example:

    test:
            $(MAKE) $(AM_MAKEFLAGS) --keep-going check

Now `make test` does what I want in tests/.

Is there anything more "Automaky" I could do to achieve the same goal?
Or is using `make --keep-going check` the expected method here if I
don't want Make to stop as soon as one subdirectory test fails?

Thank you for your time,
Phil

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to avoid stopping the recursive `check` target?

Mathieu Lirzin
Hello,

Philippe Proulx <[hidden email]> writes:

> So I put a TESTS variable in a/Makefile.am, b/Makefile.am, and
> c/Makefile.am, and remove the TESTS variable from tests/Makefile.am. Now
> I can go to a/, b/, or c/ and run `make check` to test only specific
> parts of the project. However, since each individual `make check` can
> fail, now the "global" `make check` in tests/ fails as soon as one
> subdirectory fails, which is the expected behaviour of Make.
>
> My current workaround is to use `make --keep-going check` in tests/, so
> that, as per make(1):
>
>     Continue as much as possible after an error.  While the target that
>     failed, and those that depend on it, can‐ not be remade, the other
>     dependencies of these targets can be processed all the same.
>
> This seems to run all the individual `make check` and exit with
> something else than 0 if one of them fails.
>
> I can also wrap this `make --keep-going check` in a new target, for
> example:
>
>     test:
>             $(MAKE) $(AM_MAKEFLAGS) --keep-going check
>
> Now `make test` does what I want in tests/.
>
> Is there anything more "Automaky" I could do to achieve the same goal?
> Or is using `make --keep-going check` the expected method here if I
> don't want Make to stop as soon as one subdirectory test fails?

I don't know if this is more "Automaky" but for that use case I would
use a non-recursive makefile with custom targets for each subset of
tests.  Here is an untested snippet:

--8<---------------cut here---------------start------------->8---
foo_tests = foo/...
bar_tests = bar/...

TESTS = $(foo_tests) $(bar_tests)

check-foo:
        $(MAKE) $(AM_MAKEFLAGS) TESTS="$(foo_tests)"

check-bar:
        $(MAKE) $(AM_MAKEFLAGS) TESTS="$(bar_tests)"
--8<---------------cut here---------------end--------------->8---

If you really want the ability to run 'make check' from sub
directories, you can create additional makefiles containing

--8<---------------cut here---------------start------------->8---
check:
        $(MAKE) -C .. check-xxx
--8<---------------cut here---------------end--------------->8---

However I would not recommend it.

HTH,

--
Mathieu Lirzin
GPG: F2A3 8D7E EB2B 6640 5761  070D 0ADE E100 9460 4D37

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to avoid stopping the recursive `check` target?

Philippe Proulx
On Tue, Aug 8, 2017 at 5:06 PM, Mathieu Lirzin <[hidden email]> wrote:

>
> Hello,
>
> Philippe Proulx <[hidden email]> writes:
>
> > So I put a TESTS variable in a/Makefile.am, b/Makefile.am, and
> > c/Makefile.am, and remove the TESTS variable from tests/Makefile.am. Now
> > I can go to a/, b/, or c/ and run `make check` to test only specific
> > parts of the project. However, since each individual `make check` can
> > fail, now the "global" `make check` in tests/ fails as soon as one
> > subdirectory fails, which is the expected behaviour of Make.
> >
> > My current workaround is to use `make --keep-going check` in tests/, so
> > that, as per make(1):
> >
> >     Continue as much as possible after an error.  While the target that
> >     failed, and those that depend on it, can‐ not be remade, the other
> >     dependencies of these targets can be processed all the same.
> >
> > This seems to run all the individual `make check` and exit with
> > something else than 0 if one of them fails.
> >
> > I can also wrap this `make --keep-going check` in a new target, for
> > example:
> >
> >     test:
> >             $(MAKE) $(AM_MAKEFLAGS) --keep-going check
> >
> > Now `make test` does what I want in tests/.
> >
> > Is there anything more "Automaky" I could do to achieve the same goal?
> > Or is using `make --keep-going check` the expected method here if I
> > don't want Make to stop as soon as one subdirectory test fails?
>
> I don't know if this is more "Automaky" but for that use case I would
> use a non-recursive makefile with custom targets for each subset of
> tests.  Here is an untested snippet:

Not a bad idea indeed.

>
> --8<---------------cut here---------------start------------->8---
> foo_tests = foo/...
> bar_tests = bar/...
>
> TESTS = $(foo_tests) $(bar_tests)
>
> check-foo:
>         $(MAKE) $(AM_MAKEFLAGS) TESTS="$(foo_tests)"

I guess you mean:

    $(MAKE) $(AM_MAKEFLAGS) TESTS="$(foo_tests)" check

>
> check-bar:
>         $(MAKE) $(AM_MAKEFLAGS) TESTS="$(bar_tests)"

Same here?

Thank you,
Phil

> --8<---------------cut here---------------end--------------->8---
>
> If you really want the ability to run 'make check' from sub
> directories, you can create additional makefiles containing
>
> --8<---------------cut here---------------start------------->8---
> check:
>         $(MAKE) -C .. check-xxx
> --8<---------------cut here---------------end--------------->8---
>
> However I would not recommend it.
>
> HTH,
>
> --
> Mathieu Lirzin
> GPG: F2A3 8D7E EB2B 6640 5761  070D 0ADE E100 9460 4D37

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to avoid stopping the recursive `check` target?

Mathieu Lirzin
Philippe Proulx <[hidden email]> writes:

> On Tue, Aug 8, 2017 at 5:06 PM, Mathieu Lirzin <[hidden email]> wrote:
>
>>
>> --8<---------------cut here---------------start------------->8---
>> foo_tests = foo/...
>> bar_tests = bar/...
>>
>> TESTS = $(foo_tests) $(bar_tests)
>>
>> check-foo:
>>         $(MAKE) $(AM_MAKEFLAGS) TESTS="$(foo_tests)"
>
> I guess you mean:
>
>     $(MAKE) $(AM_MAKEFLAGS) TESTS="$(foo_tests)" check
>
>>
>> check-bar:
>>         $(MAKE) $(AM_MAKEFLAGS) TESTS="$(bar_tests)"
>
> Same here?

Yes indeed.   :)

--
Mathieu Lirzin
GPG: F2A3 8D7E EB2B 6640 5761  070D 0ADE E100 9460 4D37

Loading...