i've been to large rallys and small rallys, and they can be different as night and day. ROT rally here in Austin probably falls in line with what you are speaking of, as well as Sturgis, Daytona Bike Week, etc. But there are tons of smaller rallys that don't really have that 'party' vibe. for the most part, they just seem to be a gathering of bikers and motorcycle enthusiasts.
there is one rally here in Texas held in Luckenbach each October that seems to gather as diverse a group of bikes and riders that i could imagine, the Harvest Classic. for the most part, the riders don't really fall into the biker stereotype and would mostly just be considered motorcycle enthusiasts. it's categorized as a European and vintage motorcycle rally and is held on the campgrounds surrounding Luckenbach, and its participants are some of the nicest bikers i've ever been around. it's probably about as tame or peaceful a rally that i can think of for its size, probably between 500-1,000 participants, and definitely family friendly, and its main purpose is to rain money for cancer. anyone who's ever been curious about rallys but don't like the party atmosphere or the stigma of biker rallys in general should definitely check it out.