Ingame experience

This is the most important. If a unit cannot be used effectively in game, then it isnt OP (yet) and if everyone uses it then it isnt UP (yet). However, beware the culture of balance. Often a unit will go for weeks without anyone using it, and then someone discovers and exploits it. Soon everyone is using it, and it goes from being 'useless' to being OP. Be careful not to repeatedly buff an unpopular unit, it can take a long time before people come back to an unpopular unit and give it another shot. Dont be too hasty to nerf a popular unit- perhaps this is a 'fad', perhaps someone will find a counter. If you do nerf it, be conservative- dont try and over-nerf it just to ensure people stop using it (Then its useless and will get buffed again).

Balance based on what good players do. This does not mean that you should balance the game entirely around the high-micro, hard to use 'pro' units or balance so that the game is only viable at a high level. However, good or inventive players can totally change the face of the way the game is played by effectively exploiting an underused unit. There are many units which an average player will not understand the use for until hey have seen it used well.

Dont balance based on one game. You should play several, on several maps, with varying skill levels between the players. It is useful if testing a strategy to try it out vs someone, then get them to try it out vs you, both trying to counter.

1v1 is the most pure testing ground for a strategy. Team games often have different requirements but the various factors in play in team games make it very hard to get an impartial judgement.

Maths and Stats

This is an incredibly useful tool in determining where the problem in a unit might lay but more importantly in determining precisely what units you should be trying to exploit. Statistical abnormalities are excellent candidates for trying to exploit ingame.

Remember there are a huge range of values besides hp/dps. Range, speed, manoeuvrability, and even variables in the script like turret turnrate. The use of a unit is not always determined by good stats, but often by the right combination and values. There is a huge difference between just outranging an LLT, and having the same range as an LLT. Try to think more in terms of unit relationships than pure numbers. When balancing a unit, try and think of other areas that can be changed to make it useful, rather than just its hp/dps. What specifically is holding it back/making it too strong ingame? Also remember knock-on effects. If you nerf something, another unit which it has until now effectively countered may suddenly be OP.

Also, see MathematicalBalancing.

Closed testing

Do for-cost tests of units, micro'ing both sides with another player or by pausing and switching teams. This can help to nullify a whole range of factors that, like economic differences and terrain. Note however that those factors are often precisely why those units are op, and that this is useful merely as a guide to get an idea of where problems might be.

All these options should generally be explored when attempting to balance a unit.

Keep units unique

If a unit is overpowered, the first thought is to nerf its strong point. That leads to dull, generalist units. Find a weakness and increase the severity of it, or make the unit more specialized.