That is honestly where I started. I didn't want you to just go there as you seemed interested in the castle bits.
I found his wall sections to not be do-able for me as they all need a topper and the included topper is a house so I built my own from a rigid foam insulation board and put it on little wooden base and used enough of the paper towers and stuff to make a castle. the paper towers are also a wide enough base that they alone as a nice card stock can hold all of the troops you can fit into them but the narrower wall sections topple over very easily and make a game very difficult. Youll need to fill them with something and a rigid foam comes into mind.
Then, after you build a castle you will still need a boat load of everything else from the siege equipement list.

not to mention probably twice as much infantry in your army

Once I realized I would not be able to find all of this stuff on ebay I would need to make it myself and just went with everything as custom. I plan on replacing my wall sections with a resin casting by myself later on but my castle works really well right now if you ask me. And it was all free minus the ink to print the handful of towers and houses. His gateway tower is really cool though, I love that thing, the gate is a separate piece that can slide easily in and out of the tower when you smash it down with a battering ram.
This has all taken me 2 years to build with a normal family life and with patience I have gathered up a lot of earthworks and siegeworks from ebay.