I have been listening to a lot of Boulez lately and actually feel some of his music is quite beautiful. I especially like Le Marteau sans Maitre, Sur Incises, Messagiques, Notations for Orchestra, and Pli Selon Pli. Some of his later work sounds like atonal Debussy; rather than relying on traditional melody and harmony, I think he creates sound worlds using pitches that clash with each other and unorthodox instrumentation. Boulez, in his early days, was heavily influenced by non-European music, especially from African and Javanese music. Some of his pieces, like Le Marteau sans Maitre (which is scored for flutes, vibraphones, guitar, bongos, maracas, violas, etc) relies on a musical vocabulary that is different from traditional western music. I think many of the composers from the Boulez generation, including John Cage, Luciano Berio, Stockhausen, and others were turning away from the traditional way of writing classical music, and turning to other ways of writing music. Not all of it is good, but there was some good music written by those guys. If you have a chance to listen to Sur Incises or Le Marteau, I would be interested in reading people's thoughts.
Do you think one of the reasons Boulez, Stockhausen, Berio, etc are not frequently played is because of the level of difficulty of the music or perhaps the unusual scoring for many of the compositions? As mentioned, Le Marteau has pretty unusual scoring. Others were incorporating electronic media and so on. Most traditional orchestras would not have a guitar and bongo player at hand to play a piece like Le Marteau. I would think most community orchestras would not have the resources to gather the various instrumentation required. I believe this was one of the reasons Boulez created the Ensemble Intercontemporain--a dedicated group of players who had training and background in playing serial-type music
Other "modern" composers I like are Elliot Carter, George Crumb, Ligeti, Messiaen, Schoenberg, Varese and some John Cage. Carter wrote in the atonal mold, but I still find his music engaging, a bit like Bartok. Have you heard Berio's Sinfonia? I think this piece is fairly accessible; while not using traditional melody, it does incorporate musical motives that are not too grating on the ear. And the third movement is based on the third movement of Mahler's 2nd along with a mash up of Debussy's La Mer, Beethoven, and other composers.