Put simply, "hop utilization" refers to how well the hops have imparted their taste and aroma into the beer.
For those that are looking for a more technical definition, hop utilization is the percentage of alpha acids that have been isomerized remain in the finished beer. Usually only about half of the alpha acids are isomerized during a one hour boil. Of this half, some will precipitate with the hot and cold breaks and others will be adsorbed by yeast during fermentation.
Hop utilization can be as low as 10 percent and seldom exceeds 40 percent. Hop utilization is affected not only by the vigor of the boil, but also by its duration. Generally speaking, the longer the wort is boiled with the hops, the more of the bitter resins will be isomerized.
Another factor affecting utilization is the pH of the wort. The higher the pH, the greater the isomerization of hop resins. The best flavor is achieved when the wort pH is 5.0 to 5.4. On the other hand, a very low wort pH reduces utilization and has other pernicious effects on flavor as well as destroying your chances of forming a good hot break. The pH relies more on how the mash happens.
In a high gravity wort, the concentration of sugars acts to block isomerization. High gravity beers always suffer from this phenomenon, and must be brewed using hop rates higher than would be needed for a normal strength beer of similar bitterness.
From a formulaic or mathematical point of view, hop utilization breaks down accordingly: %U(utilization) = (isoalpha acids present¸ alpha acid used) x 100. If you are following a recipe and the desired bitterness is stated in terms of IBU's you can use the following formula to calculate how much bittering hops to use when boiling your wort. (Percentages are expressed as whole numbers, i.e., 15% = 15. Volumes are total volumes for the recipe.) In metric units: Weight(grams) = (Volume in liters x IBU x 10) , (% Utilization x % alpha acid of hops). In English units: Weight(ounces) = (Volume in gallons x IBU x 1.34), (% Utilization x % alpha acid of hops).
There are many hop calculators on the internet that can also assist you in calculations. A quick web search should produce serveral options.