Instance limitations

Count Limitations
As of Patch 1.9 or so, players are limited to entering 5 instances per hour. Meaning that if you start your first run at 1:00pm, and do 4 more runs within an hour, you will not be able to enter any more instances (aside from Battlegrounds) until it is 2:00pm. An addon like Loot & Reset can illustrate this to you and help you time your runs.

Example:
 * 1:00 : Instance 1 entered
 * 1:10 : Instance 2 entered
 * 1:20 : Instance 3 entered
 * 1:30 : Instance 4 entered
 * 1:40 : Instance 5 entered
 * 1:50 : Instance 6 Unable to enter. You will simply walk through the portal without entering the instance.
 * 2:00 : Instance 6 entered

Note: The number of instances you can enter is linked to your other characters on the same account as well. This means that if you have reached the entering limit on character A, character B will not be able to enter any instance.

Instance lock
Once you kill a boss in a heroic dungeon or raid, you receive an instance lock with respect to that boss. While the instance lock is in place, you can't enter any copy of the instance in which the boss is alive, nor can you queue for it in the Dungeon Finder. If you reset the instance by entering it alone, the new copy will be missing the defeated boss. Instance locks reset at 4:00 server time every morning for heroics and every Tuesday morning for raids.

Random heroics are exempt from instance locks: if you receive a dungeon that you are saved to as a random, you can enter it as though you weren't saved. However, random heroics do create instance locks. What this means is that in one day, you can clear a heroic dungeon first as a specific (or by traveling to it without the Dungeon Finder) and then as a random, but not vice-versa.

Player Limitations
Some instances are limited to a party of 5. These are considered dungeon, party, or just plain instances. Some others have a player limit of 10, 20, 25 or 40 players. These are considered Raid instances. Most raid instances will not allow you to enter unless you are in a raid.

Level Limitations
Instances are designed for a certain character level - mainly because of the level of the mobs inside. Higher level mobs will crit your tank more often, dodge more attacks and resist your casters' spells. Lower level mobs, on the other hand, may not pose enough of a challenge and will give you little to no experience.

The Looking For Group tool shows the instances suggested for that level.

Each instance also has a minimum required level to actually be able to zone in, however this is considerably lower than the suggested level. Both these level requirements can be checked at the Instances by level article.