[cmake-developers] Should CMAKE_LINK_DEPENDS_NO_SHARED be on by default?

[Stephen Kelly]

Hi,

CMAKE_LINK_DEPENDS_NO_SHARED was introduced in this cycle, but off by
default. It seems useful enough to be on by default. Is there any reason not
to enable it by default?

Thanks,

Steve.

[Brad King]

On 02/24/2013 10:29 AM, Stephen Kelly wrote:
> CMAKE_LINK_DEPENDS_NO_SHARED was introduced in this cycle, but off by
> default. It seems useful enough to be on by default.

For reference, it was introduced in this thread:

http://thread.gmane.org/gmane.comp.programming.tools.cmake.user/43246/focus=44469

> Is there any reason not to enable it by default?

It is a change in default behavior. The current behavior was
introduced a long time ago after some very subtle breakage
occurred due to not re-linking dependents. However, the
current behavior also makes everyone pay a cost in the common
case to support the potentially rare cases where not re-linking
is wrong. Unfortunately I do not remember enough detail about
the original case to argue about whether it could be common
(IIRC it had something to do with moving a .cxx from one shared
library to another without changing any headers).

If the option were to be on by default then instead there
should be the reverse option off by default with a name like:

CMAKE_LINK_DEPENDS_SHARED_LIBRARIES

Otherwise we will ask every target to carry around a property
setting just to get default behavior.

-Brad

[Stephen Kelly]

Brad King wrote:

> If the option were to be on by default then instead there
> should be the reverse option off by default with a name like:
>
> CMAKE_LINK_DEPENDS_SHARED_LIBRARIES

I implemented that but one of the dashboards already fails:

http://open.cdash.org/testDetails.php?test=178646760&build=2827189

I don't know if that's just because of dirty-building, or because I extended
the test, or because my logical change has a bug.

Thanks,

Steve.

[Brad King]

On 02/25/2013 11:46 AM, Stephen Kelly wrote:
> Brad King wrote:
>
>> If the option were to be on by default then instead there
>> should be the reverse option off by default with a name like:
>>
>> CMAKE_LINK_DEPENDS_SHARED_LIBRARIES
>
> I implemented that but one of the dashboards already fails:
>
> http://open.cdash.org/testDetails.php?test=178646760&build=2827189
>
> I don't know if that's just because of dirty-building, or because I extended
> the test, or because my logical change has a bug.

It is a real failure. The problem is that on Windows the dependency
goes through the import library (.lib), not the dll. The import
library does not get updated when the .dll is rebuilt because the
set of dllexport-ed symbols does not change. Therefore the _relink
executable does not really need to relink.

-Brad

[Alexander Neundorf]

On Monday 25 February 2013, Brad King wrote:
> On 02/24/2013 10:29 AM, Stephen Kelly wrote:
> > CMAKE_LINK_DEPENDS_NO_SHARED was introduced in this cycle, but off by
> > default. It seems useful enough to be on by default.
>
> For reference, it was introduced in this thread:
>
> http://thread.gmane.org/gmane.comp.programming.tools.cmake.user/43246/focus
> =44469
>
> > Is there any reason not to enable it by default?
>
> It is a change in default behavior.

Maybe with a policy ?

Alex

[Brad King]

On 02/25/2013 02:18 PM, Alexander Neundorf wrote:
>> It is a change in default behavior.
>
> Maybe with a policy ?

How would it trigger? We can't warn on every target that links
to a shared library. We're not changing any existing interfaces.

The behavior only affects the dependencies placed in the build
system. It won't affect one-shot builds, only developer workflow.
The dependencies we're removing are a common source of complaints,
and should not be necessary in normal situations.

I am on the fence about the behavior change. The solution I added
before, CMAKE_LINK_DEPENDS_NO_SHARED, has no change but allows
developers to optimize extra linking out of their workflow by
adding the setting themselves.

-Brad

[Alexander Neundorf]

On Monday 25 February 2013, Brad King wrote:
> On 02/25/2013 02:18 PM, Alexander Neundorf wrote:
> >> It is a change in default behavior.
> >
> > Maybe with a policy ?
>
> How would it trigger? We can't warn on every target that links
> to a shared library. We're not changing any existing interfaces.

if the requested version is higher, simply default to the new behaviour,
otherwise to the old.
This would be a bit different than the other policies.

Alex

[Brad King]

On 02/25/2013 02:35 PM, Alexander Neundorf wrote:
> if the requested version is higher, simply default to the new behaviour,
> otherwise to the old.
> This would be a bit different than the other policies.

So unset == OLD (internally WARN == OLD)?

If no warning will ever appear we do not need a policy at all.
We can simply define the behavior. With either old or new
default behavior though we still need an explicit option as
a property. Perhaps we should say that if

(CMAKE_)LINK_DEPENDS_SHARED_LIBRARIES

is set to true or false then it is honored. If not then the
current minimum required version of CMake determines the value.

-Brad

[Matthew Woehlke]

On 2013-02-24 10:29, Stephen Kelly wrote:
> CMAKE_LINK_DEPENDS_NO_SHARED was introduced in this cycle, but off by
> default. It seems useful enough to be on by default. Is there any reason not
> to enable it by default?

What is the use case for this?

I'm having a hard time imagining a case where it buys you anything. What
would cause a .so to change often that you would *not* want to re-link?

--
Matthew

[Brad King]

On 02/25/2013 01:29 PM, Matthew Woehlke wrote:
> On 2013-02-24 10:29, Stephen Kelly wrote:
>> CMAKE_LINK_DEPENDS_NO_SHARED was introduced in this cycle, but off by
>> default. It seems useful enough to be on by default. Is there any reason not
>> to enable it by default?
>
> What is the use case for this?
>
> I'm having a hard time imagining a case where it buys you anything. What
> would cause a .so to change often that you would *not* want to re-link?

The case here is that when a .so re-links its *dependents* do not
need to be re-linked too. Since the .so is not statically linked
into the dependents they should not need to be linked again. The
dynamic loader will simply see the updated .so at runtime. Refer
back to the thread I linked earlier:

http://thread.gmane.org/gmane.comp.programming.tools.cmake.user/43246/focus=44469

-Brad

[Matthew Woehlke]

On 2013-02-25 14:00, Brad King wrote:
> On 02/25/2013 01:29 PM, Matthew Woehlke wrote:
>> On 2013-02-24 10:29, Stephen Kelly wrote:
>>> CMAKE_LINK_DEPENDS_NO_SHARED was introduced in this cycle, but off by
>>> default. It seems useful enough to be on by default. Is there any reason not
>>> to enable it by default?
>>
>> What is the use case for this?
>>
>> I'm having a hard time imagining a case where it buys you anything. What
>> would cause a .so to change often that you would *not* want to re-link?
>
> The case here is that when a .so re-links its *dependents* do not
> need to be re-linked too. Since the .so is not statically linked
> into the dependents they should not need to be linked again. The
> dynamic loader will simply see the updated .so at runtime. Refer
> back to the thread I linked earlier:
>
> http://thread.gmane.org/gmane.comp.programming.tools.cmake.user/43246/focus=44469

I guess I'm still not seeing it without rereading the entire thread.
Anyway, thanks for the quick explanation.

I guess my personal opinion is that I can think of enough theoretical
ways where a relink might be needed, and haven't encountered a project
where the relink cost is sufficiently high, to feel that I would
personally want to not relink.

If CMake could somehow compare the public API that was previously
linked, versus the new flavor thereof, I would probably be more willing
to trust such an optimization. (One could also argue that this would be
a better job for the build tool.)

--
Matthew

[Brad King]

On 02/25/2013 02:56 PM, Matthew Woehlke wrote:
> I guess my personal opinion is that I can think of enough theoretical
> ways where a relink might be needed, and haven't encountered a project
> where the relink cost is sufficiently high, to feel that I would
> personally want to not relink.

You've just explained the reason CMake has works the way it does now ;)
However, there have been complaints from people with projects that
have hundreds of targets depending on a few shared libraries. A
change to one of the core libraries takes a long time to re-link
everything unnecessarily.

The question is whether we should make everyone pay the cost to
relink until they discover that a new CMake allows them to disable
it, or make everyone take the risk of not relinking until they
discover they need to set an option to play it safe.

Can you elaborate on some of the theoretical cases where relinking
will be needed but no header files have changed? It would be useful
to have them available for discussion.

-Brad

[d***@aol.com]

> Can you elaborate on some of the theoretical cases where relinking
> will be needed but no header files have changed? It would be useful
> to have them available for discussion.


I can think of one, but it’s probably not that common:



A header file declares a function prototype, but there is no implementation of it.



When it’s discovered, when somebody finally tries to add an implementation, you only have the change the C++ file to add it.



Of course, for the caller that discovers this, it had never linked successfully in the first place, and would relink after that rebuild anyhow...



But for other callers that had never discovered it, though, a relink may be necessary in case the ordinals of some of the *other* functions in the dll changed due to the addition of one in the middle.



Uncommon, but possible.

[Brad King]

On 02/25/2013 03:09 PM, ***@aol.com wrote:
>
>> Can you elaborate on some of the theoretical cases where relinking
>> will be needed but no header files have changed? It would be useful
>> to have them available for discussion.
> I can think of one, but it’s probably not that common:
>
> A header file declares a function prototype, but there is no implementation of it.
>
> When it’s discovered, when somebody finally tries to add an implementation, you only have the change the C++ file to add it.
>
> Of course, for the caller that discovers this, it had never linked successfully in the first place, and would relink after that rebuild anyhow...
>
> But for other callers that had never discovered it, though, a relink may be necessary in case the ordinals of some of the *other* functions in the dll changed due to the addition of one in the middle.
>
> Uncommon, but possible.

Actually for DLL platforms the dependency goes on the import library,
and some toolchains may not touch the import library if it has not
changed. Perhaps we should not make this option affect DLL platforms.
Some investigation into Windows toolchain behavior may be needed.

-Brad

[Stephen Kelly]

Brad King wrote:

> On 02/25/2013 03:09 PM, ***@aol.com wrote:
>>
>>> Can you elaborate on some of the theoretical cases where relinking
>>> will be needed but no header files have changed? It would be useful
>>> to have them available for discussion.
>> I can think of one, but it’s probably not that common:
>>
>> A header file declares a function prototype, but there is no
>> implementation of it.
>>
>> When it’s discovered, when somebody finally tries to add an
>> implementation, you only have the change the C++ file to add it.
>>
>> Of course, for the caller that discovers this, it had never linked
>> successfully in the first place, and would relink after that rebuild
>> anyhow...
>>
>> But for other callers that had never discovered it, though, a relink may
>> be necessary in case the ordinals of some of the *other* functions in the
>> dll changed due to the addition of one in the middle.
>>
>> Uncommon, but possible.
>
> Actually for DLL platforms the dependency goes on the import library,
> and some toolchains may not touch the import library if it has not
> changed. Perhaps we should not make this option affect DLL platforms.
> Some investigation into Windows toolchain behavior may be needed.
>

As the DLL platforms (and Mac for a reason I don't see
http://open.cdash.org/testDetails.php?test=176688935&build=2827249) are
failing, I've reverted the patches for now so as to not turn the dashboard
all red tomorrow.

Thanks,

Steve.

[Matthew Woehlke]

On 2013-02-25 15:02, Brad King wrote:
> Can you elaborate on some of the theoretical cases where relinking
> will be needed but no header files have changed? It would be useful
> to have them available for discussion.

The possibility that first came to mind is where the API depends on
compiler flags or similar preprocessor-ish bits, especially if these are
not well guarded with something like a configured config.h to change
when these change (and ensure that users get the same imports as what
was built).

For example, turning PIC on/off, changing the language support level,
etc. might cause the API to change in a way that is not reflected in the
headers (especially language support level, since most projects don't
check for that... although conversely one can also hope that wouldn't
cause an API break...).

A slightly contrived example would be if you are relying on other than
the CMake default-defined symbol to detect when your own library is
being built, and forget to define it (or accidentally undefine it), such
that all of the sudden there is a large change in your export set.

Another example is running a system upgrade across a mass rebuild, such
that system libraries are suddenly linked with a different compiler
version than before. Depending on what else has changed, and how the
system package manager works, I could see that this *might* not cause
the time stamps on the header files to change.

I wasn't thinking of it, but the example by dlrdave is also a good one,
and in my experience not all that wildly unusual... only applies to
lazy-resolving platforms, however (i.e. not Windows).

To some extent, I suppose it comes down to how one weighs correctness
versus speed. For most projects, the speed hit is probably small, so I'd
be inclined to favor correctness, and recommend large projects that know
that the cost for them is unusually high to override the choice.

--
Matthew

[Brad King]

On 2/25/2013 5:18 PM, Matthew Woehlke wrote:
> The possibility that first came to mind is where the API depends on
> compiler flags or similar preprocessor-ish bits, especially if these are
> not well guarded with something like a configured config.h to change
> when these change (and ensure that users get the same imports as what
> was built).
>
> For example, turning PIC on/off, changing the language support level,
> etc. might cause the API to change in a way that is not reflected in the
> headers (especially language support level, since most projects don't
> check for that... although conversely one can also hope that wouldn't
> cause an API break...).
>
> A slightly contrived example would be if you are relying on other than
> the CMake default-defined symbol to detect when your own library is
> being built, and forget to define it (or accidentally undefine it), such
> that all of the sudden there is a large change in your export set.
>
> Another example is running a system upgrade across a mass rebuild, such
> that system libraries are suddenly linked with a different compiler
> version than before. Depending on what else has changed, and how the
> system package manager works, I could see that this *might* not cause
> the time stamps on the header files to change.
>
> I wasn't thinking of it, but the example by dlrdave is also a good one,
> and in my experience not all that wildly unusual... only applies to
> lazy-resolving platforms, however (i.e. not Windows).
>
> To some extent, I suppose it comes down to how one weighs correctness
> versus speed. For most projects, the speed hit is probably small, so I'd
> be inclined to favor correctness, and recommend large projects that know
> that the cost for them is unusually high to override the choice.

These are all good examples. Of course since CMake already does
the safe thing people would not have hit these.

Steve, I think we should just leave CMAKE_LINK_DEPENDS_NO_SHARED
as it is. If you want that behavior by default in KDE then set
it in a high-level project's package configuration file.

-Brad

[Stephen Kelly]

Brad King wrote:

> On 2/25/2013 5:18 PM, Matthew Woehlke wrote:
>> The possibility that first came to mind is where the API depends on
>> compiler flags or similar preprocessor-ish bits, especially if these are
>> not well guarded with something like a configured config.h to change
>> when these change (and ensure that users get the same imports as what
>> was built).
>>
>> For example, turning PIC on/off, changing the language support level,
>> etc. might cause the API to change in a way that is not reflected in the
>> headers (especially language support level, since most projects don't
>> check for that... although conversely one can also hope that wouldn't
>> cause an API break...).
>>
>> A slightly contrived example would be if you are relying on other than
>> the CMake default-defined symbol to detect when your own library is
>> being built, and forget to define it (or accidentally undefine it), such
>> that all of the sudden there is a large change in your export set.
>>
>> Another example is running a system upgrade across a mass rebuild, such
>> that system libraries are suddenly linked with a different compiler
>> version than before. Depending on what else has changed, and how the
>> system package manager works, I could see that this *might* not cause
>> the time stamps on the header files to change.
>>
>> I wasn't thinking of it, but the example by dlrdave is also a good one,
>> and in my experience not all that wildly unusual... only applies to
>> lazy-resolving platforms, however (i.e. not Windows).
>>
>> To some extent, I suppose it comes down to how one weighs correctness
>> versus speed. For most projects, the speed hit is probably small, so I'd
>> be inclined to favor correctness, and recommend large projects that know
>> that the cost for them is unusually high to override the choice.
>
> These are all good examples. Of course since CMake already does
> the safe thing people would not have hit these.
>
> Steve, I think we should just leave CMAKE_LINK_DEPENDS_NO_SHARED
> as it is. If you want that behavior by default in KDE then set
> it in a high-level project's package configuration file.

Fair enough. I'd reverted the branch already and now I've deleted it from
stage. The unit test change is probably still useful, but would need a WIN32
branch in the test and would need to figure out what happened to the mac
dashboard I linked to before.

Thanks,

Steve.