It's honestly very easy to get into fanfiction and never break into the exchange scene at all, especially not the old-school multifandom type mostly run on Ao3 and Dreamwidth, like Yuletide. Personally, I only stumbled across it as a thing I could actually do because an older friend who'd been in the exchange scene for much longer introduced me to it one year. Before that, I sometimes saw the tag "Yuletide" on a fic in a small fandom I liked, but had no idea what it was or that I could be part of it. I imagine many other younger/new people who are joining fandom and get interested in fanfic have similar experiences, and might at best get into single-fandom exchanges and Big Bangs and such, instead...
Re: Why isn't Yuletide growing?
Personally, I only stumbled across it as a thing I could actually do because an older friend who'd been in the exchange scene for much longer introduced me to it one year. Before that, I sometimes saw the tag "Yuletide" on a fic in a small fandom I liked, but had no idea what it was or that I could be part of it. I imagine many other younger/new people who are joining fandom and get interested in fanfic have similar experiences, and might at best get into single-fandom exchanges and Big Bangs and such, instead...