The most prestigious think tank in the world, the Brookings Institution, once annoyed then president Richard Nixon so much that he proposed fire-bombing its headquarters.

One of his aides, John W. Dean III, described how in recordings from the Oval Office Nixon can be heard, "literally pounding on his desk, saying, 'I want that break-in at the Brookings'".

He figured that during the blaze his operatives might be able to sneak past the firefighters to steal files from its safe, Dean later told The Washington Post.

The fire-bombing never happened and Nixon later resigned in disgrace (over other scandals), but Brookings has thrived. It was named think tank of the year in 2018 by the University of Pennsylvania's Think Tank and Civil Societies Program, a kind of think tank think tank.

Though think tanks are constantly cited and quoted in media, and often even in policy documents, confusion remains about what exactly it is that they do, why and how they do it, and for whom.

According to the Think Tank and Civil Societies Program's 2018 report, there were 8248 think tanks in the world last year, including 42 in Australia. The highest-ranked Australian outfit was the Australian Institute for International Affairs, which came in at number 56 in the world, followed by the Lowy Institute for International Policy at 64. The ranking is the result of 1796 interviews with peer institutions and experts from the print and electronic media, academia, public and private donor institutions and governments around the world.

So what do these think tanks do? And what effect do they have?

What's the difference between a think tank and a lobby group?

Typically a lobby group acts as a hired gun, prosecuting the interests of its members – which may be major commercial interests – while a think tank conducts and presents the research of its scholars and experts.

In practice, the line becomes blurred, especially when think tanks undertake work for clients on commission.

Some think tanks in Australia are viewed by critics as acting like lobby groups, pushing industry-friendly research or junk science in support of interest groups.

There are other distinctions. According to the University of Pennsylvania, think tanks are structured as permanent bodies rather than as ad hoc commissions or research panels; and they devote a substantial portion of their resources to commissioning and publishing research and policy analysis in the social science areas such as political science, economics, public administration and international affairs.

How do think tanks work?

The Lowy Institute was founded in 2003 with a gift from shopping-centre mogul Frank Lowy and his family and instructions to academic Michael Fullilove, who wrote the original feasibility study, to help create an organisation that would "deepen the debate in Australia about the world, and give Australia a greater voice abroad".

The billionaire's philanthropic gesture vaulted the Lowy Institute to the front rank of Australian think tanks when it launched.

Today it functions as a model think tank. Its experts produce original research in fields run by program directors who report to research director Alex Oliver and Fullilove, who serves as an executive director answering to a board made up of Lowy family members as well as business, political, academic and retired military heavyweights.

Think tank directors generally agree that their key role is to bridge the gap between other sources of information – such as media and academia – and policymakers.

Ben Oquist, director of the Australia Institute, says that even the best academic research can be slow to produce and difficult to digest, while journalists often produce timely information that is not deeply enough researched to inform policymaking.

The work of a think tank takes up the middle ground.

Normally, that research is made freely available to anyone who wants to use it and it is often presented in talks and conferences hosted by the institutions. The Lowy Institute publishes its flagship papers as Penguin Paperbacks and some of its most significant research is crafted as digital interactives that compare aid spending and diplomatic influence in Asia and the Pacific. This information is used widely in diplomatic, political, military and media circles around the region and the world.

Who sets a think tank's agenda?

Most think tanks are transparent about their core values. Both the Labor and Liberal parties run affiliated think tanks to inform their policymaking, though these think tanks' influence surges and recedes over the years.

Because they are expensive to run, think tanks can have a tendency to amplify the voices of those individuals and organisations – public or private – that fund them.

According to Oquist, fundraising takes up a substantial amount of time that could better be spent on research and the task has become harder since the global financial crisis. As a result large donations, either corporate or philanthropic, are crucial.

The right-leaning Institute for Public Affairs (IPA) has been accused of championing policies to reduce the wages bills, taxes and regulations on some of its biggest donors, including tobacco companies and Gina Rinehart, the chairman of Hancock Prospecting. Federal MP Tim Wilson, a former policy director at the IPA, says the suggestion that corporate donors shape the IPA's policy outlook is unfounded as its public positions remain in line with its values. "Look at the consistency of our work, there is no inconsistency,” he says.

In 2016, The New York Times published an investigation critical of leading US think tanks, which it said had allowed their research and even their internal appointments to be influenced by donations from foreign governments and corporations. Elizabeth Warren, the Democratic Party senator who is now in the running to be president, said at the time: "This is about giant corporations who figured out that by spending, hey, a few tens of millions of dollars, if they can influence outcomes here in Washington, they can make billions of dollars.”

Martin Indyk, the Australian-born diplomat and academic who was then serving as Brookings' executive vice-president (and who is now on the Lowy Institute's board) said at the time: "We do not compromise our integrity. We maintain our core values of quality, independence, as well as impact.”

Asked why the Centre for Independent Studies does not publish details about its funding, its executive director, Tom Switzer, says public debate has become so toxic that to do so would be to invite a needless barrage of criticism upon the organisation's supporters.

Fullilove says while the Lowy Institute's outlook is determinedly international, the views of its scholars are their own and they often disagree with each other. "There are no house positions," says Fullilove. "We are host to a range of opinions, but the advocate of none." He says what influence Lowy has it retains through the quality of its research and by positioning itself as the chosen venue for powerful people to introduce big ideas. This year Scott Morrison became the third sitting prime minister to deliver the Lowy Institute's annual address.

How much, and what kind of, influence do they have?

In his book Masters of the Universe: Hayek, Friedman, and the Birth of Neoliberal Politics, Daniel Stedman Jones traces the ascendancy of neoliberal politics in the West after World War II. He argues that politicians in the US and Britain came to accept the primacy of free markets, deregulation and limited government not by chance, but due to the work of a network of advocates supported and informed by think tanks founded for the purpose, particularly the Atlas Foundation and the Mont Pelerin Society.

He quotes correspondence between two of the intellectual fathers of the movement, Friedrich Hayek and Antony Fisher, in which Fisher writes: "You mentioned 'luck'! … No doubt, luck is important … Was there not an intention on both our parts and consequent action? [to start a think tank] How much is luck?"

In Australia, Switzer says the liberal reforms prosecuted by the Hawke, Keating and Howard governments were championed by the work of the Centre for Independent Studies.

In Political Troglodytes and Economic Lunatics, author Dominic Kelly argues far-right ideologues in Australia used think tanks to lend credibility to dubious research to shift public debate and government policy. His book, its title a quote from Hawke himself, argues that a small handful of men led by conservative business leader Ray Evans were able to shift Australian political thinking on industrial relations, constitutional change, Indigenous affairs and climate change to the hard right through the creation of a clutch of think tanks, some funded by the mining sector – namely, the H.R. Nicholls Society, the Samuel Griffith Society, the Bennelong Society and the Lavoisier Group.

The IPA is known for the influential roles its members have secured outside the organisation in support of its mission. Tim Wilson advocated for free speech reforms – which critics fear may serve to protect racial vilification – in line with IPA positions as the Australian Human Rights Commissioner from 2014 to 2016, before he was elected to Parliament. Other IPA current or recent members or staff who do or have served in Parliament include James Patterson, David Leyonhjelm and Bob Day. Its director, John Roskam, has twice sought Liberal Party preselection.

Fullilove notes that the reforms and changes advocated by think tanks need not be ideologically driven or partisan. In 2009 and 2011, the Lowy Institute published research showing that the reach of Australia's diplomatic network was falling behind that of comparable OECD and G20 nations.

When Julie Bishop became foreign minister in 2013 she ordered a review of Australia's diplomatic footprint and two years later announced that five new overseas missions would be opened – the largest expansion of our diplomatic network in 40 years. She said that the Institute's research had been "ringing in her ears" when she ordered the review. "Since Bishop's announcement,” says Fullilove, "10 posts have opened and a further five have been promised in the Pacific.”

Wilson says that the IPA has done much to support sometimes controversial free speech laws and extend freedom of religion protections.

When are think tanks most effective?

Oquist says think tanks are at their best when they dramatically shift stale policy thinking. He says the political consensus that Australia needs a federal anti-corruption commission was driven in part by Australia Institute research and advocacy, as was the acceptance that superannuation tax concessions needed to be reined in. He says the Australia Institute was able to shift the needle on these issues by presenting compelling evidence of the benefits of change consistently over a prolonged period.

Trent Hagland, an adjunct professor at the University of Sydney who is undertaking doctoral studies on the role and impact of think tanks, agrees, saying that think tanks have the luxury of long-term thinking that political parties no longer enjoy. In think tanks, he says, unpopular ideas can be kept alive over decades rather than election cycles.

"Look at the debate over nuclear weapons,” he says, referring to the suggestion by ANU professor Hugh White this year that Australia might need to consider their development in the possible absence of US power in the Pacific.

"No one was talking about nuclear weapons before he wrote about them in his book. Now ASPI [the Australian Strategic Policy Institute] is discussing it.

"It would be politically toxic for any party to go to an election today proposing to develop nuclear weapons. But over time, if the public debate is held for years, the public becomes desensitised to it.”

Or, as Oquist puts it, think tanks are most successful when they "make the radical seem reasonable”.

This explainer was suggested by subscriber Peter Fleming, who was curious to know more about the think tanks that influence politics and public opinion.