I have run into issues like MFC is not supported - does that mean I can't build Windows apps?
No, it just means that you cannot build them with MFC when using an Express version of Visual Studio. You may be able to get MFC support in some other way for free, but I have no idea as I do not use MFC.
And I've not been used to writing console apps before....
You have not written any C++ programs before, so that is not very important.
So, by using "Express", have I limited myself and should I look to a full function C++.
No, the compiler that comes with the Express version of Visual Studio is an optimising compiler that is mostly standard conformant and with pretty much full support for the standard library.
And if so, is there a freebie version or do I have to purchase C++.
You cannot purchase C++, no more than you can purchase English

Should I stay clear of Express ?
It is fine for most novices and hobbyists.
If you want to write programs that utilise a graphical user interface, then you have other options besides MFC, including these cross platform libraries:
FLTK,
FOX,
gtkmm,
Qt,
wxWidgets.