Hedonism is overrated – to make the best of life there must be pain, says this Yale professor

Hedonism is the simplest theory of human nature. The nature of suffering and pain is to be avoided. The spirit of this view is captured in The Epic of Gilgamesh. The destiny of men is to make merry each day, dance and play. The Canadian rock band Trooper wrote a song called "We're here for a good time, Not a long time, So have a good time, The sun can't shine every day."

Hedonists wouldn't deny that life is full of voluntary suffering - we wake up in the middle of the night to feed the baby, take the 8.15 into the city and undergo painful medical procedures. The unpleasant acts are seen by the hedonist as the costs that must be paid to get better pleasures in the future. Challenging and difficult work is the ticket to survival and status; boring exercise and unpleasant diet are what you have to go through for a vibrant old age.

There is something right here. We possess drives for food, sex, status and much more, and that much of our suffering is chosen with these ends in mind.

Hedonism is an awful theory. My latest book, The Sweet Spot: Suffering, Pleasure, and the Key to a Good Life, makes the case for a different theory of what people want. I argue that we also want to live meaningful lives, and this involves willingly experiencing pain, anxiety, and struggle. We see value in suffering.

People willingly climb mountains, run marathons, or get punched in the face in gym and dojos. Young men who choose to go to war are hoping to experience challenge, fear and struggle, to be christened by fire, to use the cliché phrase. Some of us choose to have children, and usually we have a sense of how hard it will be; maybe we even know of all the research showing that, moment by moment, the years with young children can be more stress than any other time of life.

We often choose to suffer. The first Matrix was designed to be a perfect human world, which was shown in the movie The Matrix, where Agent Smith tells Morpheus how the world they are experiencing is a simulation created by malevolent computers. Everyone would be happy there. It was a disaster. Entire crops were lost when no one would accept the programme. Some believed that we didn't have the programming language to describe our perfect world, but I believe that we do. The perfect world was a dream your primitive cerebrum kept trying to wake up from.

Pain and struggle are part of what we want.

Why would we choose to suffer? Hedonist would tell you that sometimes it is for the sake of tangible goals. Pain can help distract us from our fears. It is possible to serve social goals by choosing to suffer. Fear and sadness are unpleasant emotions that can provide moral satisfaction. In the right contexts, effort and struggle can lead to the joys of mastery and flow.

There is more. The economist gave an example of mountaineers. It seems that the pleasures here are not obvious, but rather that it seems to belenting misery from end to end. There is a constant craving for food, and diaries and journals by climbers talk about it. On a typical ascent, the vast majority of time is spent in mind-bogglingly boring activities, such as beingweathered out for many hours in a small smelly tent crammed in with other climbers. Climbers describe their experiences as lonely and alienating, spending days and weeks in bitter silence, with disagreements that don't get smoothed over. People do it, and then do it again and again, getting some satisfaction that doesn't diminish in any way.

For some of us, a life well lived is more than a life of pleasure and happiness. I agree with the economist who wrote that what is good about an individual human life can't be boiled down to any single value. It's not all about beauty or justice. A variety of relevant values, including human wellbeing, justice, fairness, beauty, the artistic peaks of human achievement, the quality of mercy, and the many different and, indeed, sometimes contrasting kinds of happiness, can be found in pluralistic theories. Life is not easy.

There is a desire to do meaningful things. Life feels incomplete if this motivation is not fulfilled. A typical reaction to finding meaning in one's life is to say "before I started school striking I had no energy, no friends and I didn't speak to anyone." I was alone at home with an eating disorder. Since I have found a meaning in a world that sometimes seems meaningless to so many people, all of that is gone.

The conclusion was similar by Viktor Frankl. Frankl studied depression and suicide in Vienna in the 1930s. Austria was taken over by the Nazis in 1938. Frankl was one of the millions of Jews who ended up in a concentration camp because he refused to abandon his patients or his elderly parents. Frankl studied his fellow prisoners and wondered what distinguishes those who maintain a positive attitude from those who can't.

The answer is meaning. Those who had the best chance of survival were those who had a reason to live. Those who have a reason to live can bear with almost any way.

Frankl was interested in mental health. His plea for a life of meaning was a central part of the therapy he developed after leaving the camps. This is the kind of existence we should want to pursue, he believed. He was sensitive to the distinction between happiness and eudaemonia, but he also referred to flourishing in a more general sense. Frankl believed that eudaemonia mattered.

People with meaning tend to be those who flourish.

How do we make meaning to be suffering? There is a lot of evidence suggesting a connection. People who say their lives are meaningful report more worry and struggle than people who say their lives are happy. Poor countries tend to be the ones where citizens report the most meaning. The countries with the happiest people are more prosperous and safe. People say that being a medical professional or a member of the clergy is the most meaningful job because it involves dealing with other people's pain. When asked to describe the most meaningful experiences of our lives, we tend to think of those on the extremes.

We don't seek out suffering. We look for meaning and purpose. Difficulties include anxiety, stress, conflict, boredom, and often physical and emotional pain. We know that training for a marathon, raising children, and climbing Everest are the things that matter the most to us.

Wouldn't a life without suffering be boring? Alan Watts is the British philosopher and popular interpreter of Zen Buddhism.

Watts asks you to imagine that you can dream about whatever you want. You could have a dream that lasted 75 years if you had this power. What would you do? He says you would choose every type of pleasure. It would be a party.

If you can do it again the next night, you can do it again the next day. Watts says you would say to yourself, "Now let's have a surprise, a dream which isn't under control, where something is gonna happen to me but I don't know what it's gonna be."

Adding more risk, uncertainty, and deprivation would make you gamble more. You would put obstacles in your way until you could finally dream of living the life you are actually living.

Is your life the best that it can be? Probably not. Watts has a fantasy that is close to the truth.

The Sweet Spot: Suffering, Pleasure and the Key to a Good Life was published by Bodley Head. You can buy it at guardianbookshop.com.