More control over 'make dist'

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

More control over 'make dist'

Michal Privoznik
Dear list,

I'm a libvirt devel and I've ran into interesting problem. I'd like to
hear your opinions on it.

Libvirt is a virtualization library that uses XML to store a virtual
machine config. We have couple of tests in our repository too that check
whether XML configs are valid, and whether some operations change it in
expected way. Some operations don't change the XML at all, in which case
we just symlink the output file to point to the input file:

ln -s $xml.in $xml.out

However, I was looking into archive produced by 'make dist' the other
day and found out that the symlinks are not preserved. I've traced down
the problem and found that autoconf is just hardcoding some of tar's
options. Namely -chf. Yes, it is -h that causes a symlink to be
dereferenced.

So my question is, what do you think of making -h configurable? We could
add new tar-* option to AM_INIT_AUTOMAKE, say tar-symlinks, which would
suppress -h on tar's command line.

What are your thoughts?

Michal

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: More control over 'make dist'

Peter Rosin
On 2016-09-14 11:33, Michal Privoznik wrote:

> Dear list,
>
> I'm a libvirt devel and I've ran into interesting problem. I'd like to
> hear your opinions on it.
>
> Libvirt is a virtualization library that uses XML to store a virtual
> machine config. We have couple of tests in our repository too that check
> whether XML configs are valid, and whether some operations change it in
> expected way. Some operations don't change the XML at all, in which case
> we just symlink the output file to point to the input file:
>
> ln -s $xml.in $xml.out
>
> However, I was looking into archive produced by 'make dist' the other
> day and found out that the symlinks are not preserved. I've traced down
> the problem and found that autoconf is just hardcoding some of tar's
> options. Namely -chf. Yes, it is -h that causes a symlink to be
> dereferenced.
>
> So my question is, what do you think of making -h configurable? We could
> add new tar-* option to AM_INIT_AUTOMAKE, say tar-symlinks, which would
> suppress -h on tar's command line.
>
> What are your thoughts?

I believe that is for the benefit of supporting unpacking the release
tarball on systems that do not support symlinks, or where symlinks are
not as flexible as one might wish for.

Cheers,
Peter

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: More control over 'make dist'

Warren Young-2
On Sep 14, 2016, at 7:49 AM, Peter Rosin <[hidden email]> wrote:

>
> On 2016-09-14 11:33, Michal Privoznik wrote:
>>
>> ln -s $xml.in $xml.out
>>
>> However, I was looking into archive produced by 'make dist' the other
>> day and found out that the symlinks are not preserved.
>
> I believe that is for the benefit of supporting unpacking the release
> tarball on systems that do not support symlinks, or where symlinks are
> not as flexible as one might wish for.

The question, then, is whether libvirt would ever be unpacked on such a system.

I’m barely aware of what libvirt does, but I think I can come up with a plausible scenario: libvirt built from source on Cygwin.

It appears from the home page that libvirt already supports Hyper-V, so a naive user might decide to build it under Cygwin rather than whatever native Windows toolchain is currently used for that case.  (The reason being that libvirt, coming from the Linux world, probably builds better under a Unix-like environment.)  Since NTFS symlinks have a number of unfortunate limitations[1] and restrictions[2], the tarball almost certainly won’t unpack correctly.

[1] https://en.wikipedia.org/wiki/NTFS_symbolic_link#Restrictions
[2] https://en.wikipedia.org/wiki/NTFS_symbolic_link#Limitations
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: More control over 'make dist'

Eric Blake-3
[re-adding Michal; not sure he is a subscriber. It's usually best to
reply-to-all rather than assume that all readers are on-list]

On 09/14/2016 02:10 PM, Warren Young wrote:

> On Sep 14, 2016, at 7:49 AM, Peter Rosin <[hidden email]> wrote:
>>
>> On 2016-09-14 11:33, Michal Privoznik wrote:
>>>
>>> ln -s $xml.in $xml.out
>>>
>>> However, I was looking into archive produced by 'make dist' the other
>>> day and found out that the symlinks are not preserved.
>>
>> I believe that is for the benefit of supporting unpacking the release
>> tarball on systems that do not support symlinks, or where symlinks are
>> not as flexible as one might wish for.
>
> The question, then, is whether libvirt would ever be unpacked on such a system.
>
> I’m barely aware of what libvirt does, but I think I can come up with a plausible scenario: libvirt built from source on Cygwin.
>
> It appears from the home page that libvirt already supports Hyper-V, so a naive user might decide to build it under Cygwin rather than whatever native Windows toolchain is currently used for that case.  (The reason being that libvirt, coming from the Linux world, probably builds better under a Unix-like environment.)  Since NTFS symlinks have a number of unfortunate limitations[1] and restrictions[2], the tarball almost certainly won’t unpack correctly.
Cygwin unpacks and handles symlinks just fine as one of the many things
it emulates, so that's not the issue (note, however, that non-cygwin
apps are generally unable to recognize cygwin symlinks, which are
intentionally NOT done as NTFS symlinks because NTFS symlinks are not
POSIXy).

I'd be more worried if trying to unpack libvirt and build it with mingw,
where you no longer have cygwin symlinks in the mix, and would indeed be
limited to just NTFS symlinks.  But I think that mingw ports of tar
already take care of that, by turning symlinks in the tarball into file
copies (where the link target was also in the tarball) or an error
message (where the link target is unknown) (although I haven't actually
tested what MSYS would actually do, so I welcome further comments from
anyone with experience).

And yes, libvirt is (cross-)built for mingw on Fedora already, as well
as me getting ready to prepare a Cygwin distribution build of libvirt
(both for the remote machine control aspect, and for the HyperV aspect).
 I'm less familiar with cross-building for mingw to know for certain
whether symlinks in tarballs cause problems for mingw targets, or
whether Wine emulation on Linux even handles symlinks.  So that agrees
with the notion of making tarballs as portable as possible by avoiding
symlinks.

On the other hand, maybe the GNU Coding Standards should be updated to
state that avoiding symlinks is no longer required of GNU packages (it's
entirely plausible that when automake first coded up the tar -h rule,
years ago, there really were Unixy systems where either the file system
or the tar program didn't handle symlinks as gracefully as desired,
which is a different beast than today's mingw situation).

--
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


signature.asc (617 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: More control over 'make dist'

Daniel Herring
In reply to this post by Michal Privoznik
Hi Michal,

I've always handled this by distributing the single real file, along with
a script that creates missing symlinks later, either during configure or
as part of one of the make targets.

Its a bit quirky, but not too annoying.  Since the dist target is for
making source packages, there are plenty of natural places to hide this.

Many related pieces of infrastructure, including Debian packages, have
the same basic symlink packing issue and workaround.

- Daniel


On Wed, 14 Sep 2016, Michal Privoznik wrote:

> Dear list,
>
> I'm a libvirt devel and I've ran into interesting problem. I'd like to
> hear your opinions on it.
>
> Libvirt is a virtualization library that uses XML to store a virtual
> machine config. We have couple of tests in our repository too that check
> whether XML configs are valid, and whether some operations change it in
> expected way. Some operations don't change the XML at all, in which case
> we just symlink the output file to point to the input file:
>
> ln -s $xml.in $xml.out
>
> However, I was looking into archive produced by 'make dist' the other
> day and found out that the symlinks are not preserved. I've traced down
> the problem and found that autoconf is just hardcoding some of tar's
> options. Namely -chf. Yes, it is -h that causes a symlink to be
> dereferenced.
>
> So my question is, what do you think of making -h configurable? We could
> add new tar-* option to AM_INIT_AUTOMAKE, say tar-symlinks, which would
> suppress -h on tar's command line.
>
> What are your thoughts?
>
> Michal
>
>

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: More control over 'make dist'

Michal Privoznik
In reply to this post by Eric Blake-3
On 14.09.2016 21:35, Eric Blake wrote:
> [re-adding Michal; not sure he is a subscriber. It's usually best to
> reply-to-all rather than assume that all readers are on-list]

Thank you, I'll use your e-mail to reply to previous ones too.

>
> On 09/14/2016 02:10 PM, Warren Young wrote:
>> On Sep 14, 2016, at 7:49 AM, Peter Rosin <[hidden email]> wrote:
>>>
>>> On 2016-09-14 11:33, Michal Privoznik wrote:
>>>>
>>>> ln -s $xml.in $xml.out
>>>>
>>>> However, I was looking into archive produced by 'make dist' the other
>>>> day and found out that the symlinks are not preserved.
>>>
>>> I believe that is for the benefit of supporting unpacking the release
>>> tarball on systems that do not support symlinks, or where symlinks are
>>> not as flexible as one might wish for.

Well, I believe it's project developers to say whether they want to
support it or not. For instance if there's a project implementing a
linux kernel module I think it's fair to expect some reasonable
environment too. Moreover there are just a few filesystems that don't
support symlinks (including FAT which would be dead already if it wasn't
for all the portable electronics).

>>
>> The question, then, is whether libvirt would ever be unpacked on such a system.
>>
>> I’m barely aware of what libvirt does, but I think I can come up with a plausible scenario: libvirt built from source on Cygwin.
>>
>> It appears from the home page that libvirt already supports Hyper-V, so a naive user might decide to build it under Cygwin rather than whatever native Windows toolchain is currently used for that case.  (The reason being that libvirt, coming from the Linux world, probably builds better under a Unix-like environment.)  Since NTFS symlinks have a number of unfortunate limitations[1] and restrictions[2], the tarball almost certainly won’t unpack correctly.

Well, we've stopped providing official .exe a long ago (2011), but we
still allow users to compile client library using mingw (no tests are
run there though). But still - it is build under Linux with some
reasonable filesystem (i.e. one supporting symlinks).

>
> Cygwin unpacks and handles symlinks just fine as one of the many things
> it emulates, so that's not the issue (note, however, that non-cygwin
> apps are generally unable to recognize cygwin symlinks, which are
> intentionally NOT done as NTFS symlinks because NTFS symlinks are not
> POSIXy).
>
> I'd be more worried if trying to unpack libvirt and build it with mingw,
> where you no longer have cygwin symlinks in the mix, and would indeed be
> limited to just NTFS symlinks.  But I think that mingw ports of tar
> already take care of that, by turning symlinks in the tarball into file
> copies (where the link target was also in the tarball) or an error
> message (where the link target is unknown) (although I haven't actually
> tested what MSYS would actually do, so I welcome further comments from
> anyone with experience).
>
> And yes, libvirt is (cross-)built for mingw on Fedora already, as well
> as me getting ready to prepare a Cygwin distribution build of libvirt
> (both for the remote machine control aspect, and for the HyperV aspect).
>  I'm less familiar with cross-building for mingw to know for certain
> whether symlinks in tarballs cause problems for mingw targets, or
> whether Wine emulation on Linux even handles symlinks.  So that agrees
> with the notion of making tarballs as portable as possible by avoiding
> symlinks.
>
> On the other hand, maybe the GNU Coding Standards should be updated to
> state that avoiding symlinks is no longer required of GNU packages (it's
> entirely plausible that when automake first coded up the tar -h rule,
> years ago, there really were Unixy systems where either the file system
> or the tar program didn't handle symlinks as gracefully as desired,
> which is a different beast than today's mingw situation).
>

Agreed. I'm not advocating for every project out there using it, I just
think that it would be nice if developers are given the choice.

Michal

Loading...