[cmake-developers] How to handle dependencies of protobuf files ?

[Alexander Neundorf]

Hi,

I stumbled upon a problem with protobuf files, I attached a testcase.
There is a MyBase.proto, which is "imported" by Complex.proto.
If MyBase.proto is modified, protoc is run again in MyBase.proto, but not on
Complex.proto, although it should.
You can have a look at the attached example.

The message MyData (in Complex.proto) has a member MyBase b1.
If I rename the message MyBase (in MyBase.proto) e.g. to MyBaseXYZ, then the
build fails, because Complex.pb.h was not regenerated, so it still refered to
the now not existing class MyBase.

Is there already a solution to handle this ?

I think to do it properly, there would have to be a dependency scanning for
proto files like there is for C/C++ headers.
Parsing at the proto-files at cmake time wouldn't be good enough (since editing
a proto file doesn't trigger a cmake run).

Comments ?

Alex

[Alexander Neundorf]

Any comments ?

Alex

Post by Alexander Neundorf
Hi,
I stumbled upon a problem with protobuf files, I attached a testcase.
There is a MyBase.proto, which is "imported" by Complex.proto.
If MyBase.proto is modified, protoc is run again in MyBase.proto, but not on
Complex.proto, although it should.
You can have a look at the attached example.
The message MyData (in Complex.proto) has a member MyBase b1.
If I rename the message MyBase (in MyBase.proto) e.g. to MyBaseXYZ, then the
build fails, because Complex.pb.h was not regenerated, so it still refered
to the now not existing class MyBase.
Is there already a solution to handle this ?
I think to do it properly, there would have to be a dependency scanning for
proto files like there is for C/C++ headers.
Parsing at the proto-files at cmake time wouldn't be good enough (since
editing a proto file doesn't trigger a cmake run).
Comments ?
Alex

--
Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake-developers

[Brad King]

Post by Alexander Neundorf
I think to do it properly, there would have to be a dependency scanning for
proto files like there is for C/C++ headers.

In order to handle implicit dependencies like that implied by

import "MyBase.proto";

then they would somehow need to be extracted from source content.
Ideally protoc should be able to write a depfile as a side effect.

Otherwise all dependencies should be listed explicitly somewhere.

-Brad
--
Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake-developers

[Alexander Neundorf]

Hi,

Post by Brad King

Post by Alexander Neundorf
I think to do it properly, there would have to be a dependency scanning for
proto files like there is for C/C++ headers.

In order to handle implicit dependencies like that implied by
import "MyBase.proto";
then they would somehow need to be extracted from source content.
Ideally protoc should be able to write a depfile as a side effect.

Parsing them using cmake would more or less work, also the include dirs are
known, so technically this would probably work. But the parsing would happen
at cmake-time, not at compile-time, but editing a proto-file does not trigger a
cmake run...
Would that have to be implemented similar to the C dependency scanning ?

Post by Brad King
Otherwise all dependencies should be listed explicitly somewhere.

so the cheap solution would be to add an argument to PROTOBUF_GENERATE_CPP()
to list proto-files these proto-files depend on which is forwarded to (every)
add_custom_command() call inside PROTOBUF_GENERATE_CPP().

Not very elegant, but at least it would make it work correctly.

This would require that if in a set of proto-files there are different
dependencies, for each of those with different dependencies a separate
PROTOBUF_GENERATE_CPP() would be needed.
A bit ugly, but better than now.

Alex
--
Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake-developers

[Brad King]

Post by Alexander Neundorf

Post by Brad King
In order to handle implicit dependencies like that implied by
import "MyBase.proto";
then they would somehow need to be extracted from source content.

Would that have to be implemented similar to the C dependency scanning ?

Yes, but even better would be if we can ask protoc to print the dependencies
out for us so we don't have to parse the sources ourselves.

Post by Alexander Neundorf
so the cheap solution would be to add an argument to PROTOBUF_GENERATE_CPP()
to list proto-files these proto-files depend on

Yes, that would be a good intermediate solution.

-Brad
--
Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake-developers

[Eric Noulard]

Post by Brad King

Post by Alexander Neundorf

Post by Brad King
In order to handle implicit dependencies like that implied by
import "MyBase.proto";
then they would somehow need to be extracted from source content.

Would that have to be implemented similar to the C dependency scanning ?

Yes, but even better would be if we can ask protoc to print the dependencies
out for us so we don't have to parse the sources ourselves.

protoc can already do something like that but it spits out a makefile
includable file.

see --dependency_out=FILE option.
Write a dependency output file in the format
expected by make. This writes the transitive
set of input file paths to FILE

moreover the generated makefile depends on the langage generator used
(--cpp_out, --java_out, --python_out, ...) because dependencies are
expressed between proto and
generated source files.

May be it would be possible to write a protoc "plugin"
https://www.expobrain.net/2015/09/13/create-a-plugin-for-google-protocol-buffer/

which would spit out easy to digest dep spec for CMake.

Unfortunately I'm not volunteering :-( just giving some idea.

Post by Brad King

Post by Alexander Neundorf
so the cheap solution would be to add an argument to

PROTOBUF_GENERATE_CPP()

Post by Alexander Neundorf
to list proto-files these proto-files depend on

Yes, that would be a good intermediate solution.
-Brad
--
Powered by www.kitware.com
http://www.cmake.org/Wiki/CMake_FAQ
Kitware offers various services to support the CMake community. For more
CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
https://cmake.org/mailman/listinfo/cmake-developers

[Brad King]

protoc can already do something like that but it spits out a makefile includable file.
see --dependency_out=FILE option.

That may work well for the Ninja generator at least. See the
add_custom_command `DEPFILE` option.

One day it would be nice to teach the Makefile generator to use
depfile-style dependencies (on compilers that support it) instead
of doing its own scanning.

-Brad
--
Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/m