Public speaking anxiety is a common social phobia in the general population, particularly among young adults. This social skill is essential for professional integration. However, few solutions have been developed to train people in public speaking. The research has shown that Virtual Reality tools can be used to improve public speaking performance. To vary the difficulty of the training, virtual audiences expressing different social attitudes may be simulated. However, only few research works have explored the impact of audience's gender on the user experience. In this article, we propose to investigate the impact of simulated social attitudes and of the gender of the virtual audience on subjective measures reflecting the user experience during a public speaking task. We describe an experimental study in which 41 participants were asked to speak in front of a virtual audience with a neutral, positive or negative social attitude. The gender of the virtual audience varied: only males, only females, or mixed-gender agents. After each speech, users assessed the audience's attitudes, self-report their emotions and their ease of public speaking in front of the audience. The results reveal that an audience with a positive attitude and composed of at least one female virtual agent is perceived as more positive and induces more positive emotions.