The Temporal Lobe!
The temporal lobes are located in the areas around the right and left ears. Not surprisingly, they deal with sound recognition—everything from noise to music to speech. This is where the famous Broca’s area and Wernicke’s areas exist, although typical language recognition and usage is usually mostly only relevant on the left side in most humans. The right side of the brain is typically more 'musical'. Because of this, it does play a role in communication too! It is important for understanding things such as prosody.

Broca’s area is associated with speech reproduction (speaking skills), while Wernicke’s area is associated with speech comprehension (understanding skills). Damage to those two areas produce speech and comprehension impairments, respectively. (Sometimes Broca's area is considered to be a part of the frontal lobe, and 'just' adjacent to the temporal lobe, rather than being a part of it.)

Again, without the PFC these skills are not very useful, so it must be stressed that the temporal lobes alone are not responsible for listening and speaking skills. The PFC and various other areas of the brain are significantly involved in the processing and development of listening and speaking skills.
Stacks Image 915
Stacks Image 909
Stacks Image 1416