My first trip to the playa kicked my ass. Not that I was in horrible physical condition - I regularly hike, walk to work, do my own gardening, etc... The altitude, heat, and dryness combined to sap my energy. I decided to be in better shape from then on.
Burning Man attracts people in a broad spectrum of ages, health, and so forth. Although there is really no minimum physical fitness requirement, I will say that the more physically fit you are the more you will enjoy your time on the playa.
As previously mentioned, conditions are a bit on the harsh side in regards to the environment - day time heat, high altitude (4,000 feet above sea level), night time cold, wind, and dust. As a bare minimum you should be prepared to deal with those conditions.
The benefits of being fairly fit extend beyond how your body deals with the extremes of weather though. You will need to walk or ride a bicycle to get around the city. Even if you walk a lot in the default world, consider that the city is approximately 3.5 miles in diameter before shrugging this off. Furthermore the ground surface varies a lot, ranging from hard packed to deep dust.
Although being in peak physical condition would be nice, those of us who aren't big fans of the gym can do some stuff to be better prepared. At a minimum, start walking a lot more in the months before the burn. If you plan to ride a bicycle, practice riding it a lot too.
If you have health issues, keep in mind that with proper planning almost anyone can attend the burn. Since I am not an expert on this, I will direct you to seek advice on your health issues both from a medical professional and for more specific advice on dealing with it on the playa seek the answer on Eplaya and on the Burning Man Web Site Health & Safety Section.