Integrating flexc++(1) and bisonc++(1) into Makefile.am

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Integrating flexc++(1) and bisonc++(1) into Makefile.am

Kip Warner
Hey list,

I'm aware autoconf has M4 macros for traditional GNU flex(1) and
bison(1) implementations of POSIX lex(1) and yacc(1) respectively. I'm
also aware the manual contains information for integrating the two into
a typical autotool'd build environment.

My challenge is replicating their functionality for flexc++(1) and
bisonc++(1) in the absense of macros to make their usage easier in
Automake. These are C++ generators for a lexer and parser, analogous to
their C counterparts. Some of the available tutorials online contradict
some of the recommendations of the Autoconf / Automake documentation,
so I figured I'd ask the gurus.

What I am trying to accomplish is have Makefile.am run the two tools on
their two respective inputs and generate their source as part of the
build process of the rest of my project. This is normally done
externally like so:

    $ flexc++ Lexer.lpp
    (produces LexerBase.h, Lexer.h, Lexer.ih, and LexerBase.cpp)

    $ bisonc++ Parser.ypp
    (produces ParserBase.h, Parser.h, Parser.ih, and Parser.cpp)

    $ g++ Parser.cpp LexerBase.cpp Main.cpp -o MyProgram

All of the generated files I consider derived for the purpose of my
project and I don't mind if the 'clean' target removes them. I figured
as I refine my grammar rules, the lexer and parser code will change a
lot anyways.

In trying to integrate the two tools into my build environment, I've
attempted the following in Makefile.am:

    ...

    BUILT_SOURCES = \
      Source/LexerBase.h \
      Source/Lexer.h \
      Source/Lexer.ih \
      Source/LexerBase.cpp \
      Source/ParserBase.h \
      Source/Parser.h \
      Source/Parser.ih \
      Source/Parser.cpp

    myprogram_SOURCES = \
      ... \
      Source/LexerBase.cpp \
      Source/Parser.cpp

    ...

    EXTRA_DIST = \
      ... \
      Source/Lexer.lpp \
      Source/Parser.ypp

    ...

    # Generate lexer source from regular expression token descriptions via flexc++...
    Source/LexerBase.h:
    Source/LogicLexer.h:
    Source/LogicLexer.ih:
    Source/LexerBase.cpp: Source/LogicLexer.lpp
       $(FLEXCPP) --target-directory=$(top_builddir)/Source $< ; \
       $(SED) -i '/#include "Lexer.ih"/a #include "ParserBase.h" \/\/ Token declarations...' $(top_builddir)/Source/LexerBase.cpp

    # Generate parser source from Backus-Naur grammar rules via bisonc++...
    Source/ParserBase.h:
    Source/Parser.h:
    Source/Parser.ih:
    Source/Parser.cpp: Source/Parser.ypp
       $(BISONCPP) --target-directory=$(top_builddir)/Source $<

FLEXCPP and BISONCPP are obtained via AC_PATH_PROG in configure.ac.

This all works ok, but I suspect this is not an elegant solution and
there are some very good suggestions from this mailing list.

Yours truly,

--
Kip Warner | Senior Software Engineer
OpenPGP signed/encrypted mail preferred
http://www.thevertigo.com

signature.asc (169 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Integrating flexc++(1) and bisonc++(1) into Makefile.am

Nick Bowler
Hello,

On 7/12/17, Kip Warner <[hidden email]> wrote:
> My challenge is replicating their functionality for flexc++(1) and
> bisonc++(1) in the absense of macros to make their usage easier in
> Automake
[...]
> In trying to integrate the two tools into my build environment, I've
> attempted the following in Makefile.am:
[...]
>     BUILT_SOURCES = \
[...]
>       Source/ParserBase.h \
>       Source/Parser.h \
>       Source/Parser.ih \
>       Source/Parser.cpp
>
>     myprogram_SOURCES = \
[...]
>       Source/Parser.cpp
[...]

>     # Generate parser source from Backus-Naur grammar rules via bisonc++...
>     Source/ParserBase.h:
>     Source/Parser.h:
>     Source/Parser.ih:
>     Source/Parser.cpp: Source/Parser.ypp
>        $(BISONCPP) --target-directory=$(top_builddir)/Source $<
>
> FLEXCPP and BISONCPP are obtained via AC_PATH_PROG in configure.ac.
>
> This all works ok, but I suspect this is not an elegant solution and
> there are some very good suggestions from this mailing list.

There aren't really any "elegant" solutions.  Make handles this kind of
tool quite badly.  It is possible to get things to work but it is always
a tradeoff between flexibility of your build system and simplicity of
your rules.

If you are happy with this method then it is totally fine.  Do make
sure parallel builds work by testing them routinely (both clean and
incrementally) -- I think listing everything in BUILT_SOURCES like you
do probably "resolves" any parallelism problems here, (by reducing
opportunities for parallelism).

The Automake manual has section on writing portable make rules for tools
that produce multiple outputs[1], with a discussion of various approaches
and their limitations.  I generally prefer approaches using a dedicated
witness file.

Finally, consider whether you want to distribute the generated parser
sources.  That way your users don't need these tools installed just to
build your package.

[1] https://www.gnu.org/software/automake/manual/automake.html#Multiple-Outputs

Cheers,
  Nick

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Integrating flexc++(1) and bisonc++(1) into Makefile.am

Will Estes
You might also have a look at the flex test suite to see various ways in which automake is abused to build odd things for flex itself. There's a lot of edge cases there and you might get ideas.

On Wednesday, 12 July 2017, 10:23 am -0400, Nick Bowler <[hidden email]> wrote:

> Hello,
>
> On 7/12/17, Kip Warner <[hidden email]> wrote:
> > My challenge is replicating their functionality for flexc++(1) and
> > bisonc++(1) in the absense of macros to make their usage easier in
> > Automake
> [...]
> > In trying to integrate the two tools into my build environment, I've
> > attempted the following in Makefile.am:
> [...]
> >     BUILT_SOURCES = \
> [...]
> >       Source/ParserBase.h \
> >       Source/Parser.h \
> >       Source/Parser.ih \
> >       Source/Parser.cpp
> >
> >     myprogram_SOURCES = \
> [...]
> >       Source/Parser.cpp
> [...]
> >     # Generate parser source from Backus-Naur grammar rules via bisonc++...
> >     Source/ParserBase.h:
> >     Source/Parser.h:
> >     Source/Parser.ih:
> >     Source/Parser.cpp: Source/Parser.ypp
> >        $(BISONCPP) --target-directory=$(top_builddir)/Source $<
> >
> > FLEXCPP and BISONCPP are obtained via AC_PATH_PROG in configure.ac.
> >
> > This all works ok, but I suspect this is not an elegant solution and
> > there are some very good suggestions from this mailing list.
>
> There aren't really any "elegant" solutions.  Make handles this kind of
> tool quite badly.  It is possible to get things to work but it is always
> a tradeoff between flexibility of your build system and simplicity of
> your rules.
>
> If you are happy with this method then it is totally fine.  Do make
> sure parallel builds work by testing them routinely (both clean and
> incrementally) -- I think listing everything in BUILT_SOURCES like you
> do probably "resolves" any parallelism problems here, (by reducing
> opportunities for parallelism).
>
> The Automake manual has section on writing portable make rules for tools
> that produce multiple outputs[1], with a discussion of various approaches
> and their limitations.  I generally prefer approaches using a dedicated
> witness file.
>
> Finally, consider whether you want to distribute the generated parser
> sources.  That way your users don't need these tools installed just to
> build your package.
>
> [1] https://www.gnu.org/software/automake/manual/automake.html#Multiple-Outputs
>
> Cheers,
>   Nick
>

--
Will Estes
[hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Fwd: Integrating flexc++(1) and bisonc++(1) into Makefile.am

djerius
In reply to this post by Kip Warner
---------- Forwarded message ----------
From: Diab Jerius <[hidden email]>
Date: Wed, Jul 12, 2017 at 10:44 AM
Subject: Re: Integrating flexc++(1) and bisonc++(1) into Makefile.am
To: Kip Warner <[hidden email]>


On Wed, Jul 12, 2017 at 12:13 AM, Kip Warner <[hidden email]> wrote:

> Hey list,
>
> I'm aware autoconf has M4 macros for traditional GNU flex(1) and
> bison(1) implementations of POSIX lex(1) and yacc(1) respectively. I'm
> also aware the manual contains information for integrating the two into
> a typical autotool'd build environment.
>
> My challenge is replicating their functionality for flexc++(1) and
> bisonc++(1) in the absense of macros to make their usage easier in
> Automake. These are C++ generators for a lexer and parser, analogous to
> their C counterparts. Some of the available tutorials online contradict
> some of the recommendations of the Autoconf / Automake documentation,
> so I figured I'd ask the gurus.
>
> What I am trying to accomplish is have Makefile.am run the two tools on
> their two respective inputs and generate their source as part of the
> build process of the rest of my project. This is normally done
> externally like so:
>
>     $ flexc++ Lexer.lpp
>     (produces LexerBase.h, Lexer.h, Lexer.ih, and LexerBase.cpp)
>
>     $ bisonc++ Parser.ypp
>     (produces ParserBase.h, Parser.h, Parser.ih, and Parser.cpp)
>
>     $ g++ Parser.cpp LexerBase.cpp Main.cpp -o MyProgram
>
> All of the generated files I consider derived for the purpose of my
> project and I don't mind if the 'clean' target removes them. I figured
> as I refine my grammar rules, the lexer and parser code will change a
> lot anyways.



Take a look at section 18.2 of the automake manual "Handling new file
extensions"

Essentially you want to tell automake to tell make that you have a
rule to create a .cpp file from a .lpp file.

As an example, here's what I have to compile Lua code

SUFFIXES += .lc .lua
AX_V_LUAC = $(AX_V_LUAC_@AM_V@)
AX_V_LUAC_ = $(AX_V_LUAC_@AM_DEFAULT_V@)
AX_V_LUAC_0 = @echo LUAC $@;
.lua.lc :
        $(AX_V_LUAC)test "$(LUA_PATH)set" = set || export
LUA_PATH="$(LUA_PATH)" ;\
        $(LUAC) -o $@ $<

It uses the automake silent rules convention to quiet things; see
section 21.3 "How Automake can help in silencing make"

It doesn't work with the automake subdir-objects option; I'm trying to
figure out why.








> In trying to integrate the two tools into my build environment, I've
> attempted the following in Makefile.am:
>
>     ...
>
>     BUILT_SOURCES = \
>       Source/LexerBase.h \
>       Source/Lexer.h \
>       Source/Lexer.ih \
>       Source/LexerBase.cpp \
>       Source/ParserBase.h \
>       Source/Parser.h \
>       Source/Parser.ih \
>       Source/Parser.cpp
>
>     myprogram_SOURCES = \
>       ... \
>       Source/LexerBase.cpp \
>       Source/Parser.cpp
>
>     ...
>
>     EXTRA_DIST = \
>       ... \
>       Source/Lexer.lpp \
>       Source/Parser.ypp
>
>     ...
>
>     # Generate lexer source from regular expression token descriptions via flexc++...
>     Source/LexerBase.h:
>     Source/LogicLexer.h:
>     Source/LogicLexer.ih:
>     Source/LexerBase.cpp: Source/LogicLexer.lpp
>             $(FLEXCPP) --target-directory=$(top_builddir)/Source $< ; \
>             $(SED) -i '/#include "Lexer.ih"/a #include "ParserBase.h" \/\/ Token declarations...' $(top_builddir)/Source/LexerBase.cpp
>
>     # Generate parser source from Backus-Naur grammar rules via bisonc++...
>     Source/ParserBase.h:
>     Source/Parser.h:
>     Source/Parser.ih:
>     Source/Parser.cpp: Source/Parser.ypp
>             $(BISONCPP) --target-directory=$(top_builddir)/Source $<
>
> FLEXCPP and BISONCPP are obtained via AC_PATH_PROG in configure.ac.
>
> This all works ok, but I suspect this is not an elegant solution and
> there are some very good suggestions from this mailing list.
>
> Yours truly,
>
> --
> Kip Warner | Senior Software Engineer
> OpenPGP signed/encrypted mail preferred
> http://www.thevertigo.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Integrating flexc++(1) and bisonc++(1) into Makefile.am

Kip Warner
In reply to this post by Nick Bowler
On Wed, 2017-07-12 at 10:23 -0400, Nick Bowler wrote:
> Hello,

Hey Nick.

> There aren't really any "elegant" solutions.  Make handles this kind
> of tool quite badly.  It is possible to get things to work but it is
> always a tradeoff between flexibility of your build system and
> simplicity of your rules.

I had suspected something along those lines due to the scarcity of
documentation that's well defined on this particular class of problem,
but feel better knowing my hack might be close to as good as it's going
to get for the time being.

> If you are happy with this method then it is totally fine.  Do make
> sure parallel builds work by testing them routinely (both clean and
> incrementally) -- I think listing everything in BUILT_SOURCES like
> you do probably "resolves" any parallelism problems here, (by
> reducing opportunities for parallelism).

Tested. It appears to work fine. Everything else will presumably be
able to build in parallel, but the flexc++(1) and bisonc++(1) derived
targets will have to be in serial.

> The Automake manual has section on writing portable make rules for
> tools that produce multiple outputs[1], with a discussion of various
> approaches and their limitations.  I generally prefer approaches
> using a dedicated witness file.

I took a look at this and I think it's a good idea. But I also think
that it may not be worth it if I have to integrate these two tools in
only one place in the build environment, as opposed to with multiple
parsers in multiple sub projects or some such.

> Finally, consider whether you want to distribute the generated parser
> sources.  That way your users don't need these tools installed just
> to build your package.

I think that as long as I don't have to actually modify the generated
parser sources, I don't have a problem requiring the user to have the
two tools installed. Most of the compilation I'll be doing, and most of
what remains will probably be on a build server which will
automatically pull the required build dependencies (e.g. sbuild debs).

Thanks a lot for your help, Nick.

--
Kip Warner | Senior Software Engineer
OpenPGP signed/encrypted mail preferred
http://www.thevertigo.com

signature.asc (169 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Integrating flexc++(1) and bisonc++(1) into Makefile.am

Kip Warner
In reply to this post by Will Estes
On Wed, 2017-07-12 at 10:59 -0400, Will Estes wrote:
> You might also have a look at the flex test suite to see various ways
> in which automake is abused to build odd things for flex itself.
> There's a lot of edge cases there and you might get ideas.

Great idea. Thanks Will.

--
Kip Warner | Senior Software Engineer
OpenPGP signed/encrypted mail preferred
http://www.thevertigo.com

signature.asc (169 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Fwd: Integrating flexc++(1) and bisonc++(1) into Makefile.am

Kip Warner
In reply to this post by djerius
On Wed, 2017-07-12 at 11:55 -0400, Diab Jerius wrote:
> Take a look at section 18.2 of the automake manual "Handling new file
> extensions"
>
> Essentially you want to tell automake to tell make that you have a
> rule to create a .cpp file from a .lpp file.

Intuitively this seems to make sense. My issue is that I don't have
hundreds of .lpp files, and only anticipate one for the entire source
tree. Further, there's a little ad hoc magic I've got to do on the only
one in running its generated .cpp through sed to insert some code. If
you have an elegant suggestion for this, I'm all ears.

> As an example, here's what I have to compile Lua code
>
> SUFFIXES += .lc .lua
> AX_V_LUAC = $(AX_V_LUAC_@AM_V@)
> AX_V_LUAC_ = $(AX_V_LUAC_@AM_DEFAULT_V@)
> AX_V_LUAC_0 = @echo LUAC $@;
> .lua.lc :
>         $(AX_V_LUAC)test "$(LUA_PATH)set" = set || export
> LUA_PATH="$(LUA_PATH)" ;\
>         $(LUAC) -o $@ $<
>
> It uses the automake silent rules convention to quiet things; see
> section 21.3 "How Automake can help in silencing make"
Excellent idea for extending the silent rules. I was actually going to
ask you guys that next. Now I just wish colorgcc recognized it ;)

--
Kip Warner | Senior Software Engineer
OpenPGP signed/encrypted mail preferred
http://www.thevertigo.com

signature.asc (169 bytes) Download Attachment
Loading...