This paper describes a psychoacoustic test in the Exterior Effects Room (EER) at the NASA Langley Research Center. The test investigated the degree to which sound quality metrics (sharpness, tonality, etc.) are predictive of annoyance to notional sounds of Urban Air Mobility (UAM) vehicles (e.g., air taxis). A suite of 136 unique (4.6 second duration) UAM rotor noise stimuli was generated. These stimuli were based on aeroacoustic predictions of a NASA reference UAM quadrotor aircraft under two flight conditions. The synthesizer changed rotor noise parameters such as the blade passage frequency, the relative level of broadband self-noise, and the relative level of tonal motor noise. With loudness constant, the synthesis parameters impacted sound quality in a way that created a spread of predictors both in synthesizer parameters and in sound quality metrics. Forty subjects listened to the suite of UAM noise stimuli in the EER and judged each sound individually on a standard scale of annoyance. Additionally, a subset of the UAM noise stimuli were compared to a reference sound that varied in loudness. From these responses, the relative effect of changes in loudness or changes in other sound quality metrics on annoyance was evaluated. This paper covers background and motivation for the test, details of how the sound stimuli were generated, and details of the test design and execution. Test results investigate how sound quality may affect perceived annoyance to UAM vehicle noise, indicating the importance of sharpness, tonality, impulsiveness, and roughness on annoyance to UAM noise.