Generally speaking, dairy products don't fall into a strict Paleo/Primal/hunter-gatherer/caveman diet because they were not consumed before the advent of agriculture.
Imagine trying to sneak up on a wild bison to casually squeeze milk out of its teats... probably not a great survival strategy!
You can see why dairy wasn't a viable food choice before agriculture and the domestication of wild beasts. Much better to kill the bison from a distance and eat its meat than take a hoof to the face with your side of bison cream.
With the rise of civilization dairy began to play a significant role in many societies, particularly in Northern Europe where many of us can trace back our lineage.
The fact that up to 95% of Northern Europeans now have lactase persistence - i.e. the ability to metabolise lactose after weaning and into adulthood - goes to show that a relatively rapid evolutionary adaptation has taken place in the last 10,000 years or so (Kresser, 2012).
This would suggest that the ability to derive nutrition from this novel food source gave our neolithic ancestors a survival advantage. Hence the evolutionary pressure to propagate the lactase persistence gene was great and the gene mutation caught on quickly.
In my opinion it is naive to take the position that since dairy was not available to our hunter-gatherer ancestors we shouldn't consume it today. This is typical of the flawed Paleo logic that any neolithic (modern) foods are inherently bad and should be avoided. I assure you, die-hard Paleoistas, that coconut oil, almond butter, bacon and sweet potato fries are all very new foods and were not available to our Paleo ancestors, even if they do fit in with a modern "Paleo template" and get talked about ad nauseam at CrossFit gyms across the world.
So it seems that there has been a fairly rapid adaptation to dairy consumption. Yet the questions still remains, is dairy good or bad for us? Well, like most things nutrition, it is both. Let's first look at the shortfalls of milk and dairy consumption and when to avoid it.
Still today roughly 65 percent of the world's population cannot tolerate lactose, especially those of Asian decent (Cordain, 2014). If you are one such person, or even if you aren't necessarily lactose intolerant but do not handle dairy very well - for example your skin breaks out, you get bad gas, bloating or diarrhea, or if dairy hinders your weight loss, then you should probably avoid it completely at least for a few months before attempting to reintroduce it.
In the past dairy has been condemned as a pro-inflammatory food. The science behind this is pretty week and the quality and types of dairy are so varied that it is impossible to categorize all dairy as being either pro-inflammatory or anti-inflammatory. It may be both, depending on the type of dairy and the person consuming it.
The components of dairy that would tend to promote inflammation are the proteins (casein and whey) and the carbohydrate (lactose). Lactose and casein, in particular, can be very allergenic and problematic to susceptible people and in this sense will contribute to inflammation.
The fats in dairy, in contrast, are generally benign and can even be anti-inflammatory.
If your body is dealing with excess inflammation - from a poor diet, chronic stress, overtraining or disease such as autoimmunity, rheumatoid arthritis, acne, etc, - then you should largely avoid lactose and casein.
Milk, for example, is very high in lactose and has many undesirable characteristics, which will be covered below.
Effect on insulin
The protein and sugars in dairy are insulinemic: meaning that they cause an acute spike in insulin release into the bloodstream (Sisson, 2011). For the most part excessive insulin release is undesirable and can lead to insulin resistance, weight gain and eventually diabetes in an unhealthy metabolism.
The effect tends to be less in full-fat dairy products. Given that the spike in insulin tends to be short lived, I don’t think the insulinemic effect of dairy needs to be a deal-breaker in healthy individuals. However, it may explain part of the reason why some people struggle to lose weight while consuming a lot of dairy.
Industrialized milk production
Pasteurization is the industrial process of flash heating milk to sterilize it. Homogenization is the industrial process of super blending milk so that the fat molecules are miniaturized and distributed evenly.
We are also in the silly habit of removing most of the fat from milk - and therefore nutrition - offering mostly low-fat or no-fat milk and dairy products, which tend to be more inflammatory and insulinemic than full fat dairy.
Additionally, factory farming operations promote an unhealthy environment for the cows - from poor corn-based diets, overuse of antibiotics (and even hormones), illness and ill-treatment. These practices all contribute to an unhealthier end product.
All of the above factors severely diminish any potential health benefits of milk and increase the negative attributes of it.
This is why I believe milk should be avoided by all folks, unless you can find raw milk and want to experiment with that.
Unfortunately raw milk, which is infinitely healthier than the bastardized milk we find in stores, is illegal in most States due to overblown concerns about pathogens.
So milk is bad, but what about other dairy products?
Healthier Dairy Options
Generally speaking, the less lactose (sugar) and casein and whey (protein) a product has then the less problematic it will be.
Full-fat dairy, particularly fermented types, from healthy sources of either raw or grass-fed, organic milk can be very nutritious foods and I personally incorporate them in my diet.
Butter and ghee (clarified butter) are basically pure fat so these tend to be well tolerated by just about everyone. I recommend grass-fed butter as a cheap, healthy, nutrient-dense and delicious fat source.
Next down the list is full-fat fermented dairy such as (Greek) yoghurt and kefir. Typically the longer the fermentation process the less lactose is remaining and the more beneficial bacteria. Kefir is particularly high in probiotics.
Hard cheese such as parmesan tend to have less lactose and more fat than soft cheeses such as mozzarella so are preferable.
Heavy cream and sour cream also have relatively low amounts of lactose and high amounts of fat so may come next in line in terms of dairy to experiment with.
Soft cheeses, ice cream, (low fat) cottage cheese and cream cheese are relatively high in lactose and casein so are less preferable in terms of their inflammatory, allergenic and insulinemic properties.
Dairy is by no means a necessary addition to the Paleo diet. For anyone starting Paleo for the first time I always recommend that they remove all dairy (except for butter) for at least the first 30 days. After which time they can experiment with the slow reintroduction of a few healthier dairy options, but only if they really want to.
If you have inflammatory conditions or are trying to lose weight I also suggest avoiding dairy until these issues are resolved.
There are no essential amino acids or fats in dairy that can't be obtained from other Paleo food sources. And contrary to popular belief your bones wont disintegrate through lack of calcium if you give up dairy completely. Dairy remains a "grey-area" Paleo food for good reason. It is problematic for many people.
Having said that dairy can be a very healthy food in the right circumstances. It is also damned delicious and adds a huge amount of flexibility to your diet if tolerated. This is why I incorporate dairy in my diet as one of my Big Four Paleo Exceptions.
Aside from butter and a small serving of kefir most days I do try to limit my consumption of dairy and instead focus on seafood, meat, eggs and vegetables, which I know to be optimal food sources for me.
I'd still rather get egg on my face than a bison's hoof...
Cordain, 2014: http://thepaleodiet.com/kefir-consumption-ill-founded-at-best/
Kresser, 2014: http://chriskresser.com/kefir-the-not-quite-paleo-superfood
Kresser, 2012: http://chriskresser.com/rhr-what-science-really-says-about-the-paleo-diet-with-mat-lalonde
Sisson, 2011: http://www.marksdailyapple.com/dairy-insulin/#axzz318bHmjgH