Works for Us!
We have two girls, ages 8 and 11, so we are four, and we love our 13' Scamp. I think a lot depends on how and where you will use the trailer. We destination camp in temperate seasons when significant rain is not in the forecast, usually 3-5 days at a time. We spend most of our time outside, including cooking and eating, so inconveniences because of the trailer's small size are minimized. (One is having to change clothes in shifts.)
However, if your plans include much camping in inclement weather, a lot of cooking and eating inside, or traveling for extended time frames, you may find the trailer shrinking as the days pass!
I also think you might want to consider whether your 4 cyl. RAV-4 will be adequate by the time you load up with passengers, food & clothes for four, etc. Even our 6 cyl. Pilot strains a bit going up long grades. Post #267 in the thread "Trailer Weights in the Real World" has a link to a spreadsheet that has some helpful data, if you haven't already seen it.