As the last few diners pay the bill in a sleepy Carluccio’s in Watford, the waiter serving me an espresso also serves the hottest piece of town gossip for centuries. “I’ve heard President Obama is coming too,” he says. “How do you know?” I ask. He winds me back, through what the wife of the friend of the brother of someone told him, to the source — a member of staff at The Grove.

Hold on? Obama at The Grove, the luxury hotel that usually hosts the England football team before games at Wembley? In Watford?

Tomorrow, the country house style retreat and spa that soothes the brows of Hugh Grant and the Beckhams will host this year’s Bilderberg Group conference, in which the world’s power brokers meet under Chatham House rules to kick around the talking points of the day. According to the Bilderberg website, it is “an annual conference designed to foster dialogue between Europe and North America”, with “off-the-record discussions about mega-trends” and the major issues facing the world. If you choose to believe the conspiracy theorists, Bilderberg is running a shadow world government away from the eyes of the media.

This year’s official participant list, released by Bilderberg in a new spirit of openness, must have been put together by a Machiavellian genius to confound the agitators. They will have expected Chancellor George Osborne, puppeteer of the British economy, but what has his Labour counterpart Ed Balls done accepting the invitation? Prince of Darkness Peter Mandelson is a shoo-in but cuddly liberal Shirley Williams too? Eric Schmidt, CEO of the all-seeing Google, clinking glasses on the terrace with internet freedom fighter Lawrence Lessig? And why Watford?

The secrecy of Bilderberg is also the fascination. Founded in 1954, it takes its name from its first venue, the Hotel de Bilderberg in Oosterbeek in the Netherlands. No minutes are taken nor press conferences held but world leaders from Tony Blair and Bill Clinton to Angela Merkel have all rubbed shoulders with business leaders and academics in this private sphere.

Officially there this weekend are the cream of the Eurocracy — Jose Barroso, president of the European Commission, former Italian prime minister Mario Monti and present Dutch prime minister Mark Rutte — as well as the bosses of BP, Goldman Sachs and defence firms EADS and BAE Systems and of course the regular Henry Kissinger.

Those are the public names. You could take a punt that Mark Carney, the new governor of the Bank of England might also appear, perhaps even former French president Nicolas Sarkozy who has been padding around London this week. Boris Johnson is conveniently on holiday but Prime Minister David Cameron inconveniently not and, with so many bigshots in Watford, can he afford not to drop in?

Watford is not a place that will have troubled the conscience of world leaders before. Even Poet Laureate John Betjeman ignored it when penning poetry about the Metroland commuter corridor. Under the armpit of the M25, it is a town on the old Grand Union Canal, ring-roaded in the 1960s and now a commercial hub with a Tesco Metro, a John Lewis, a Mecca Bingo and a Carluccio’s, a mix of old-school shops and modern aspiration. The Grove, a mile north, was once part of the estate of the Earls of Clarendon. In 1996 it turned itself into a hotel, and the land around into rolling golf courses and lovingly nurtured wild meadows.

The idea of the hotel is smart but informal, the doormen swapping top hats for trilbys and lavender-coloured sweaters. Gold and marble is out. Stencils of Ruskin poetry and playful topiary of dogs playing tennis are in. There are huge vases of orchids in the foyer, toiletries provided by the White Company, three restaurants, a swimming pool and the spa that offers a men’s facial called “age rebel” — perhaps of interest to some of the Bilderberg attendees. Tiger Woods loves the golf course. Kylie always books the Vice Presidential suite. But behind the gloss, the hotel has form: it was the venue for Google’s Zeitgeist thought leaders forum just last month.

By Monday, though, the tranquillity was evaporating slowly. The reception was swamped with reservation attempts by American activist journalists, presumably not with the intention of trying the spa treatments.

As I strolled around the grounds, looking into the trees for CCTV cameras, the atmosphere shifted. The car park, at first rows of Porsches and family cars in for a day of golf as well as the hotel, began to be supplanted with unmarked white vans. Men in black wandered around officiously, and a steel perimeter fence was slowly taking shape around the main house and its wall gardens.

It was preparation against the protests that, reflecting the conference’s status, attract the cream of agitators who began to circle as soon as the news of the venue broke. David Icke, who believes the world is run by people of a reptilian bloodline, was meant to speak at an impromptu counter-Bilderberg festival down the road, which is not allowed by local police on a technicality. Alex Jones, the activist DJ who had been trying to book into the hotel, turned up there on Monday, despite his reservation being cancelled, to argue that they were discriminating against him on grounds of political views. He was ejected, very politely, by the security team.

Bilderberg has this year accepted the opposition, even demarcating a small area of the grounds for protesters. Judging by the chorus of tweets, quite a few may turn up. It’s too far though to rubberneck at what really goes on. Inside the Ochre and Silk conference rooms the participants are due to discuss an agenda, also released by the group, of big data, Africa’s challenges, the European Union and cyberwarfare. And when Kissinger and company tire of the talk, they can wander over to the walled gardens, stroll among the herb beds and pick up a mallet and play croquet.

For the more adventurous, the walled garden also conceals an outdoor swimming pool surrounded by sun-loungers and a re-created square of sandy beach, complete with volleyball net. President Obama would probably have been great on the court. It seems terrible to dash the hopes of the residents and waiters of Watford but while there will still be a few surprise attendees, it is highly unlikely the US President will be one of them. His next European trip is later this month when he’s visiting Northern Ireland for the G8 conference before going on to Berlin for a state visit.

The conspirators may take succour, though, that they are not the only ones grasping for the details of what is going on. One person on this year’s guest list is Sherard Cowper-Coles, former diplomat and now a director at BAE, who points out that even those inside the perimeter are victims of Bilderberg’s steel ring of secrecy.

“I’m honoured to have been asked to go,” he told me last night in London. “But it’s crazy. They [the organisers] wouldn’t tell us where we were going to stay until a couple of weeks beforehand,” adding with a smirk:  “Some people had to book their private jets in advance.”