Building a Linux/FreeBSD build for a custom designed board

I have a custom board (ARM Cortex-A9 Dual Core chip) which I need to turn into a control and interface card for a machine. This entails me developing a stand alone touch (resistive touch screen on an LCD, all custom) application which allows the user to choose and set parameters and run the machine, maybe select and load profiles (stored as files, so I may need a basic filesystem) etc.

I am thinking of using Linux as the base which launches the application directly without going into a traditional desktop (I am not sure but I think GTK can do that?). I need a point of origin to start from.

I have studied this extensively on the internet, have downloaded books (Understanding Linux Kernel) and tried LFS as well. All of which have been great help. But I still need a reference point to start from.

I am thinking of using TinyCore as my starting point, but I am not sure if it will be able to configure all the attached hardware (A USB port, serial port, Ethernet, LCD and touch screen) and launch properly.

Thank for any pointers or sanity check.