Discussion:
[cmake-developers] new generator question - xml file output for embedded IDE platforms.
d***@duaneellis.com
2018-05-01 20:50:58 UTC
Permalink
Hi -

I'm looking into adding a new "generator" type, that is basically a
fancy form of "configure_file()"

At this point, I've been stepping through Cmake code trying to
understand the general flow
and want to ask the question: Is this insane or stupid? Or not a bad
idea.

Some details to understand where I am headed and what I'm thinking.

Generally, cmake produces a makefile, or - for example with Visual
Studio it produces an XML file directly.

In my case, I am focusing on micro-controller *embedded* targets - and I
need to produce various XML files that are required by IDEs.
In other cases I need to create GNU makefiles for command line gcc-arm
it varies.

I also need the ability to create Visual Studio (or linux) projects
because it is often very helpful to create unit tests for libraries that
can run on a host platform for some embedded libraries - it is this unit
test part that makes Cmake is an interesting solution.

For the EMBEDDED target - some assumptions & major compromises for this
type of target is important and must be made - ie: You cannot compile
and execute something, many of the various tests and such will just not
be possible. The IDEs often cannot really manage re-running Cmake -
(basically some IDEs perform an IMPORT operation)

To simplify - I want to limit the supported items to two things:
Build (1 to N) static libraries.
Build & Link (1 to N) applications, typically this produces an ELF
file
Optionally extract a HEX or BIN file from the ELF file.

The IDEs generally have:
A top level Workspace file - Much like a Visual Studio SLN file.
Each project has its own file - much like visual studio.
There are sometimes additional files to be created
Something for the debugger, or perhaps a "config.h" type file

The goal here is not just compiling the code but fully supporting the
IDE experience, ie: all debugger features work better if you build using
the IDE

My hunch is this:
What is really needed is a TEMPLATE language - and to some degree
that is what I am proposing and want feed back on.

Assume the following is present, ie: on the command line or via a
CMakeLists.txt file in a subdirectory
A variable with the TOOL NAME (with version number)
A variable with the CHIP NAME

Then -
Based on the TOOL & CHIP NAME - I can find (and read) other files
with more details.
For example Endian, specific ARCH ie: ARM CortexM3 vrs Atmel AVR
Chip specific information like: Size of FLASH, starting address of
FLASH, RAM, etc.

AND - a directory that contains lots of "template" files

Here are some example project files that might be generated.

ARM - Kiel uVision -

https://github.com/ARM-software/CMSIS_5/tree/develop/CMSIS/DSP/Projects/GCC
Specific examples:

https://github.com/ARM-software/CMSIS_5/blob/develop/CMSIS/DSP/Projects/GCC/arm_cortexM_math.uvoptx
Source files appear around line 3000 ... there are many of
these.

IAR project

https://github.com/ARM-software/CMSIS_5/tree/develop/CMSIS/DSP/Projects/IAR
the EWW file - is the Workspace, the EWP is the PROJECT

IAR also has something called an "argvars" file- used to set
variables used across projects within a workspace.
Example:

https://github.com/ti-simplelink/ble-sdk-210-extra/blob/master/Projects/ble/multi_role/CC26xx/IAR/multi_role.custom_argvars

IAR also has "icf" files - yet another xml file.
Key point: This file is "imported" by the IDE in a *one*way* import
step.

TI CCS - supports something called a PROJECT SPEC file
Narrowly focusing on the ARM targets TI has their own compiler, plus
they support the GCC compiler)
While the TI-CCS is an ARM-eclipse environment - they do support
eclipse project files
But that has its own list of issues when we talk about cross
compiler tools.

http://processors.wiki.ti.com/index.php/ProjectSpecs_in_CCS


https://github.com/ti-simplelink/ble_examples/blob/master/examples/rtos/CC2640R2_LAUNCHXL/bleapps/peripheral_bidirectional_audio/tirtos/ccs/peripheral_bidirectional_audio_cc2640r2lp_app.projectspec

If I narrow the supported list of features to something generally like
this:

For a given target:

a) In some cases, might need to produce a "top level workspace file"
(more below)
Might need to add a standardized variable, ie:
CMAKE_IDE_WORKSPACE_NAME or something.

b) For a target - given a list of source files - create one or more
static libraries.
- CMake has this basic input construct already

c) given a list of source files - create an 'executable' - that may link
against the above libraries (plus others that are pre-built)
- Cmake has this basic construct now.

d) Some common things, ie: A list of Include Directories, a list of
command line defines
- Cmake has this

e) The ability to GENERATE something like "foobar.h" from the file
"foobar.h.in"
- Cmake has this, with some limited supported features

f) The ability to copy a file from (a template) directory into the
"build" directory
An example might be the startup assembly language file.
- Cmake has this, via the configure_file() command.

f) What's missing is the XML file creation -
I want to really avoid writing a *CUSTOM* xml creator for each tool
and each chip
The permutations are horrible.

Only SOME of the Cmake commands would be used/required/supported, for
example
(below is an invented XML syntax that shows the types of things that
might be needed)

<xml>

@if( ${FEATURE} )@
<feature-foo-enabled-enable value="true">
@else()@
<feature-foo-enabled-enable value="false">
@endif()@

<another-method value="@${FEATURE}@">

@foreach( SOURCEFILE ${SOURCE_FILE_LIST} )@
<source> @${SOURCE_FILE}@ </source>
@endforeach()@

<linker-lib-search-dir>
@foreach( this_library ${TARGET_LIBRARY_LIST} )@
@library( GET_LIBDIR this_dirname ${this_library} )@
-L@{this_dirname}@
@endforeach()@
</linker-lib-search-dir>

<linker-libnames>
@foreach( this_library ${TARGET_LIBRARY_LIST} )@
@library( GET_LIBNAME this_libname ${this_library} )@
-l@{this_libname}@
@endforeach()@
</linker-lib-search-dir>

</xml>

=============

I think - the above would cover about 80 to 90% of the use cases.

Am I insane? and architecturally Cmake is not the way to do this?


Thanks.
--
Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake-developers
Gößwein Matthias / eeas gmbh
2018-05-02 07:28:09 UTC
Permalink
Hi Duane,

As far as i understand from your mail you want to generate project files
from CMake.

Although I would appreciate a more powerful configure_file command (e.g.
a built in generator like the mustache generator
(https://mustache.github.io/) or something like that) i must confess
that configure_file is not the right command for the feature you want.
It is used to generate some files from templates (e.g. Header Files),
actually in a simple form of exchanging placeholders with contents of a
CMake Variables.

It works like that:

The feature which you want is done by the CMake Generators, which you
can specify by the command line option -G <Generator> (or you can select
it with the cmake-gui).
If an IDE is actually not supported by CMake a generator it will have to
be implemented for that in the source code of CMake. There are
generators which generate IDE projects, where you can build it with the
IDE itselt (e.g. Visual Studio) and there are so called "Extra
generators" which generate Ninja/Makefiles and additional project files
for the IDE. The build can be done by the IDE, but it will run the
Ninja/Makefiles (e.g. Eclipse CDT generator).
https://cmake.org/cmake/help/v3.11/manual/cmake-generators.7.html

The Toolchain and the Processor (Chip) may be specified by Toolchain
Files, which can also be chosen by the cmake-gui or on the command line
with the option -T <Toolchain-File>. You may have several toolchain
files for different toolchains (e.g. one for embedded and one for PC)

Some information regarding that can be found here:
https://gitlab.kitware.com/cmake/community/wikis/doc/cmake/CrossCompiling
https://cmake.org/cmake/help/latest/manual/cmake-toolchains.7.html

When CMake is run it checks for a working compiler. To check that CMake
tries to compile a simple program. For embedded compilers it is not
always possible without some special files (e.g. linker files), so CMake
also provides a special mode for that where only a library is created
instead of an executable.
(See Variable CMAKE_TRY_COMPILE_TARGET_TYPE,
https://cmake.org/cmake/help/v3.11/variable/CMAKE_TRY_COMPILE_TARGET_TYPE.html)

Best regards,
Matthias.
--
***@eeas.at /mail
www.eeas.at /web
+43 660 1280 131 /phone

------------------------------
eeas gmbh
Technologiepark 17
4320 Perg
Austria

------------------------------
ATU67456549 /uid
FN385458a /firmenbuchnummer
landesgericht linz /firmenbuch
Post by d***@duaneellis.com
Hi -
I'm looking into adding a new "generator" type, that is basically a
fancy form of "configure_file()"
At this point, I've been stepping through Cmake code trying to
understand the general flow
and want to ask the question: Is this insane or stupid? Or not a bad
idea.
Some details to understand where I am headed and what I'm thinking.
Generally, cmake produces a makefile, or - for example with Visual
Studio it produces an XML file directly.
In my case, I am focusing on micro-controller *embedded* targets - and I
need to produce various XML files that are required by IDEs.
In other cases I need to create GNU makefiles for command line gcc-arm
it varies.
I also need the ability to create Visual Studio (or linux) projects
because it is often very helpful to create unit tests for libraries that
can run on a host platform for some embedded libraries - it is this unit
test part that makes Cmake is an interesting solution.
For the EMBEDDED target - some assumptions & major compromises for this
type of target is important and must be made - ie: You cannot compile
and execute something, many of the various tests and such will just not
be possible. The IDEs often cannot really manage re-running Cmake -
(basically some IDEs perform an IMPORT operation)
Build (1 to N) static libraries.
Build & Link (1 to N) applications, typically this produces an ELF
file
Optionally extract a HEX or BIN file from the ELF file.
A top level Workspace file - Much like a Visual Studio SLN file.
Each project has its own file - much like visual studio.
There are sometimes additional files to be created
Something for the debugger, or perhaps a "config.h" type file
The goal here is not just compiling the code but fully supporting the
IDE experience, ie: all debugger features work better if you build using
the IDE
What is really needed is a TEMPLATE language - and to some degree
that is what I am proposing and want feed back on.
Assume the following is present, ie: on the command line or via a
CMakeLists.txt file in a subdirectory
A variable with the TOOL NAME (with version number)
A variable with the CHIP NAME
Then -
Based on the TOOL & CHIP NAME - I can find (and read) other files
with more details.
For example Endian, specific ARCH ie: ARM CortexM3 vrs Atmel AVR
Chip specific information like: Size of FLASH, starting address of
FLASH, RAM, etc.
AND - a directory that contains lots of "template" files
Here are some example project files that might be generated.
ARM - Kiel uVision -
https://github.com/ARM-software/CMSIS_5/tree/develop/CMSIS/DSP/Projects/GCC
https://github.com/ARM-software/CMSIS_5/blob/develop/CMSIS/DSP/Projects/GCC/arm_cortexM_math.uvoptx
Source files appear around line 3000 ... there are many of
these.
IAR project
https://github.com/ARM-software/CMSIS_5/tree/develop/CMSIS/DSP/Projects/IAR
the EWW file - is the Workspace, the EWP is the PROJECT
IAR also has something called an "argvars" file- used to set
variables used across projects within a workspace.
https://github.com/ti-simplelink/ble-sdk-210-extra/blob/master/Projects/ble/multi_role/CC26xx/IAR/multi_role.custom_argvars
IAR also has "icf" files - yet another xml file.
Key point: This file is "imported" by the IDE in a *one*way* import
step.
TI CCS - supports something called a PROJECT SPEC file
Narrowly focusing on the ARM targets TI has their own compiler, plus
they support the GCC compiler)
While the TI-CCS is an ARM-eclipse environment - they do support
eclipse project files
But that has its own list of issues when we talk about cross
compiler tools.
http://processors.wiki.ti.com/index.php/ProjectSpecs_in_CCS
https://github.com/ti-simplelink/ble_examples/blob/master/examples/rtos/CC2640R2_LAUNCHXL/bleapps/peripheral_bidirectional_audio/tirtos/ccs/peripheral_bidirectional_audio_cc2640r2lp_app.projectspec
If I narrow the supported list of features to something generally like
a) In some cases, might need to produce a "top level workspace file"
(more below)
CMAKE_IDE_WORKSPACE_NAME or something.
b) For a target - given a list of source files - create one or more
static libraries.
- CMake has this basic input construct already
c) given a list of source files - create an 'executable' - that may link
against the above libraries (plus others that are pre-built)
- Cmake has this basic construct now.
d) Some common things, ie: A list of Include Directories, a list of
command line defines
- Cmake has this
e) The ability to GENERATE something like "foobar.h" from the file
"foobar.h.in"
- Cmake has this, with some limited supported features
f) The ability to copy a file from (a template) directory into the
"build" directory
An example might be the startup assembly language file.
- Cmake has this, via the configure_file() command.
f) What's missing is the XML file creation -
I want to really avoid writing a *CUSTOM* xml creator for each tool
and each chip
The permutations are horrible.
Only SOME of the Cmake commands would be used/required/supported, for
example
(below is an invented XML syntax that shows the types of things that
might be needed)
<xml>
@if( ${FEATURE} )@
<feature-foo-enabled-enable value="true">
@else()@
<feature-foo-enabled-enable value="false">
@endif()@
@foreach( SOURCEFILE ${SOURCE_FILE_LIST} )@
@endforeach()@
<linker-lib-search-dir>
@foreach( this_library ${TARGET_LIBRARY_LIST} )@
@library( GET_LIBDIR this_dirname ${this_library} )@
@endforeach()@
</linker-lib-search-dir>
<linker-libnames>
@foreach( this_library ${TARGET_LIBRARY_LIST} )@
@library( GET_LIBNAME this_libname ${this_library} )@
@endforeach()@
</linker-lib-search-dir>
</xml>
=============
I think - the above would cover about 80 to 90% of the use cases.
Am I insane? and architecturally Cmake is not the way to do this?
Thanks.
--
Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake-developers
d***@duaneellis.com
2018-05-02 16:17:53 UTC
Permalink
Post by Gößwein Matthias / eeas gmbh
configure_file is not the right command
Yea, it's the nearest existing item, and it only does the most
simplistic replacement that's why I use that as a basis for my example.
It is in effect, like the final last 'sed' step done by gnu
autoconfigure tools. Nothing more.
Post by Gößwein Matthias / eeas gmbh
If an IDE is actually not supported by CMake a generator it will have to be implemented for that in the source code of CMake.
yea, i'm trying to avoid that - but I can write that if required :-(

It's more then the IDE, it is also the CHIP effectively the SYSTEM

What I need is the variable data that Cmake has already and I need to be
able to tell CMake that it *cannot* run the compiler instead, all of the
information about the compiler will be provided via some Cmake script,
for example names like this, either on the command line or specified in
a Cmake file that holds alot of variables.

Cmake-Embedded-${CompilerName}.txt
Cmake-Embedded-${ChipName}.txt

Possibly:

Cmake-Embedded-${RtosName}.txt

or

Cmake-Embedded-BareMetal.txt

And packages (aka: Libraries) that you might want to use would never be
discovered and would instead be specified in some form, for example

Cmake-Embedded-Package-${PackageName}.txt

An Embedded Package (aka: A static library) provided it does not require
a specific hardware access should EASILY be re-usable in a host
environment.

Thus, a package could provide for example

Cmake-Embedded-Package-HostLib-${PackageName}

And several unit test type applications like this.

Cmake-Embedded-Package-HostTest-${PackageName}

Key thing to remember, i'm approaching this from the *embedded*side*
where most - if not all of the flexibility found on a HOST simply does
not exist, thus moving an embedded package to host is easy because the
host side is far more flexibility then the embedded.

the WIN I am looking for is the "HostLib" and "HostTest" Cmake already
provides this, why re-invent the wheel.

Other alternatives is autoconfig - which - is very Windows Unfriendly,
lots and lots of embedded types use Windows - in some cases the IDE is
only available on windows.

Bottom line:

Given the information in the Cmake-Embedded-*.txt files there is enough
information to create the embedded IDE project files which are generally
simple XML files
And that's the idea ...

Thanks for your help & comments.

-Duane.
--
Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake-developers
Alexander Neundorf
2018-05-02 19:02:55 UTC
Permalink
Post by d***@duaneellis.com
Post by Gößwein Matthias / eeas gmbh
configure_file is not the right command
Yea, it's the nearest existing item, and it only does the most
simplistic replacement that's why I use that as a basis for my example.
It is in effect, like the final last 'sed' step done by gnu
autoconfigure tools. Nothing more.
Post by Gößwein Matthias / eeas gmbh
If an IDE is actually not supported by CMake a generator it will have to
be implemented for that in the source code of CMake.
yea, i'm trying to avoid that - but I can write that if required :-(
I think you'll have to do this.
Post by d***@duaneellis.com
It's more then the IDE, it is also the CHIP effectively the SYSTEM
What I need is the variable data that Cmake has already and I need to be
able to tell CMake that it *cannot* run the compiler instead, all of the
information about the compiler will be provided via some Cmake script,
for example names like this, either on the command line or specified in
a Cmake file that holds alot of variables.
Cmake-Embedded-${CompilerName}.txt
Cmake-Embedded-${ChipName}.txt
Cmake-Embedded-${RtosName}.txt
or
Cmake-Embedded-BareMetal.txt
this exists basically.
The files cmake loads are named <OS>-<tool>-<language>.cmake.
In CMake there are e.g. Generic-SDCC-C.cmake.
This would be "BareMetal" using sdcc.
You can write your own files for your own platforms.
OS and other settings can be set up in your toolchain file.

Are you familiar with cmake cross-compiling support in CMake ?
If not, please get into this, it handles many of the issues you have.

Alex
--
Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake-developers
d***@duaneellis.com
2018-05-02 22:57:21 UTC
Permalink
Post by Alexander Neundorf
Post by d***@duaneellis.com
Post by Gößwein Matthias / eeas gmbh
If an IDE is actually not supported by CMake a generator it will have to
be implemented for that in the source code of CMake.
yea, i'm trying to avoid that - but I can write that if required :-(
I think you'll have to do this.
Hmm - Another approach is to use Python to create the IDE files
And from that python script generate CMakefiles for the unit test
situation.

Sort of backwards from where I am starting, but that said - my focus is
the Embedded side, not the host.

One thing that is never easy is a "split language" development
environment. ie: Some stuff is done in C (the generator) - and some
stuff is done in Script (ie: Cmake Language)
you don't have a debugger for Cmake, just print statements, and you are
always asking your self the question: "Should this step be in Language
A, or Language B"
Post by Alexander Neundorf
Post by d***@duaneellis.com
{duane: Various cmake examples, snip snip }
Cmake-Embedded-${CompilerName}.txt
Cmake-Embedded-${ChipName}.txt
Cmake-Embedded-${RtosName}.txt
Cmake-Embedded-BareMetal.txt
this exists basically.
[snip]
In CMake there are e.g. Generic-SDCC-C.cmake.
but that uses/requires/assumes - Make files are used to run the build.
Having the IDE execute Make - while doable, is *NOT* desired.
Post by Alexander Neundorf
Post by d***@duaneellis.com
Are you familiar with cmake cross-compiling support in CMake ?
yes, I'm currently building Cmake from source, and stepping through the
code experimenting and learning how Generators work.
For my generator I basically need to make a dummy (does nothing)
generator and let everything be done in the template

Often it seems to always comes back to the generator and what needs to
be done there. It is same problem I talked about earlier - some things
are in CmakeScript, Some are in C, and others will be in the template it
self, and where does feature(X) need to be in C, or in Cmake or in the
Template?

-Duane.
--
Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake-developers
d***@duaneellis.com
2018-05-02 23:23:40 UTC
Permalink
Post by Alexander Neundorf
Are you familiar with cmake cross-compiling support in CMake ?
yes, I'm currently building Cmake from source, ....
Often it seems to always comes back to the generator and ..
I'll add for example when it is processing languages,
CMakeCInformation.cmake (and the CXX version)
which that script assumes that it will be managing not only the
compiler, but linker and libarian

Why this way? Because in this case, there is ZERO need to do all of the
"./configure" type testing
and determination because at the end of the day, CMake will not be
executing the compiler period.
The IDE will do all of that work.

This type of change is non trivial, and I'm wondering if it is the right
approach or not.

-Duane.
--
Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake-developers
Loading...