Slightly confused about how this works with brigades.

For example.... i have a brigade made up of 2 units chaos warriors with a spawn attached on one side. My opponent charges me with a unit of knights, but he only makes contact with the chaos warriors.
Am i able to shoot at the charger if he doesn't contact my chaos spawn? or is it only if he contacts the spawn?
We been playing it that it's only if the enemy contacts the actual unit can he stand and shoot. Were we correct?