Because that "glue" holding them together adds a bit of latency.
More cores don't always equal more performance. The scheduler is constantly deciding which threads of the CPU deserve which ...