The Mini-games are divided into Tiers as well. So you could have 14 people on, 6 in the 20-40 Tier and 8 in the 60-80 Tier and still not be able to start a match. You need at least 12 qued up in your Tier before it starts.
I think you'll start to see more people hop on in the higher Tiers now. For the new players of lower levels they can do it to but they need to understand how it works so pass the knowledge on.
When they release the PvP levels and rewards system I'm sure more will be joining as well.
Our guild holds an event every Saturday at 5pm Pacific to try to get as many people qued up as possible so that we can play.
http://forums.ageofconan.com/showthread.php?t=83606