Hey Xanen, where in Seattle do you train at?
And to make this a comment not-entirely devoid of content. Rolling with people of different sizes is good. If you're a small guy though, you're going to have to live with rolling mostly with people bigger than you (hey, girls seem to get used to it, you can too, *****).
That having been said, rolling with bigger guys is likely to get you injured. If you're concerned about that you may want to look at training somewhere else where there's a more diverse set of partners.