Technically speaking, the physical heart isn't capable of thought.
The cognitive process starts in the human brain that forms within an intangible 'mind' that will shape our perceptions, views, opinions and feelings that will drive our actions.
Perhaps, in the ancient times, when Human Psychology isn't exactly a well-explored topic, people needed an explanation on the cognitive process and therefore chose the heart as a central organ for controlling the intangible mind, so to speak. Its always easier to attribute a thought, feeling or reaction to a physical, easily described human organ than to complicate things with tangibility and intangibility. Simplification, it seems, is the easiest way to digest apparently 'confusing' information.