Goal-directed behavior lowers activity in brain areas that include the medial frontal cortex, the medial and lateral parietal cortex, and limbic and paralimbic brain regions, commonly referred to as the "default network." These activity decreases are believed to reflect the interruption of processes that are ongoing when the mind is in a restful state. Previously, the nature of these processes was probed by varying cognitive task parameters, but the presence of emotional processes, while often assumed, was little investigated. With fMRI, we studied the effect of systematic variations of both cognitive load and emotional stimulus connotation on task-related decreases in the default network by employing an auditory working memory (WM) task with musical sounds. The performance of the WM task, compared to passive listening, lowered the activity in medial and lateral, prefrontal, parietal, temporal, and limbic regions. In a subset of these regions, the magnitude of decrease depended on the memory load; the greater the cognitive load, the larger the magnitude of the observed decrease. Furthermore, in the right amygdala and the left precuneus, areas previously associated with processing of unpleasant dissonant musical sounds, there was an interaction between the experimental condition and the stimulus type. The current results are consistent with the previously reported effect of task difficulty on task-related brain activation decreases. The results also indicate that task-related decreases may be further modulated by the emotional stimulus connotation.