Serum (blood) cholesterol is produced by your own liver, it's not from dietary sources. Only a small percentage of people have a greater relationship between dietary cholesterol and serum cholesterol. Most people have little or no relationship between the two. Cholesterol is vital to health. It is used by the body in the skin to produce it's own vitamin D3, which is more like a hormone than a vitamin, along with many other vital things.
The reason doctors and the medical community freak out over high cholesterol is that people with high serum cholesterol also seem to have more plaque and cholesterol deposits in their arteries, which cause blockages and lead to strokes and heart attacks. However, because the cholesterol in your blood is produced by your own liver and is the liver's response to excess carbohydrate consumption, it's not about how much dietary cholesterol you consume. The liver takes excess carbs and converts them into cholesterol, then the body uses that cholesterol in the blood to patch lesions in the artery walls that are caused by chronic high blood sugar (from insulin resistance and over consuming carbs) and high insulin levels. Both sugar in the blood, and insulin, are highly irritating to the artery walls and as one heart surgeon said, every time you eat something loaded with sugar, it's like scraping a wire brush through the insides of your arteries.
But as others have said, your total number of cholesterol doesn't matter to a certain extent. Your ratio between the different types of cholesterol are what's important.