patrick.net

#privateintelligence agencies Madame President:
Your Tuesday morning briefing...Yesterday we talked about Iranian Revolutionary Guard efforts to carry out political assassinations in the US. (As far as we know, none have been successful.) Much of this can be credited to the growth of super high technology private intelligence agencies that are not under some of the same legal constraints as state intelligence agencies. In the case of the Iranians, a private spy firm identifies the section inside the Revolutionary Guards tasked with political assassinations. They get to know every man and woman inside this section. Profiles are maintained on each person and updated in real-time. Any electronic communication from anyone inside that section is captured (including probably encrypted platforms.).
One of my most beloved books of all time is "The Dogs of War" by Frederick Forsythe. It is a novel about the planning and execution of a coup plot against a small African country. The author goes into the most excruciating detail on how the plotters avoided detection by British law enforcement and international intelligence agencies. Phones were not used. Handwritten notes and whispering in the ear sufficed. Once the Iranian Revolutionary Guard figures this out, assassination plots will become very dangerous.
I shall share with you a fascinating article from the Economist Magazine on private intelligence agencies. There is an audio section at the front of the article to make it easier for you to absorb:

Technology Quarterly | Intelligence, Inc
Private firms and open sources are giving spies a run for their money
There is plenty of co-operation, too
Illustration of an eye with a keyhole in it. The background is a motherboard.illustration: claire merchlinsky
Jul 1st 2024

Save

Share

Give
Listen to this story. Enjoy more audio and podcasts on iOS or Android.
As railways expanded across America in the 19th century, there was little law enforcement. Rail barons needed to keep track of threats. The Pinkerton detective agency frequently filled the gap, recruiting informants and passing dossiers to sheriffs. “That was the booming technology of that era,” says Andrew Borene, an American former intelligence officer. Today it is the internet. Mr Borene is executive director of Flashpoint, a “threat-intelligence” company which monitors terrorist groups and hostile intelligence services online, selling the information to governments and businesses. A few decades ago that work would have been the preserve of spy agencies.

The deluge of data is fuelling a boom in private-sector intelligence. This is empowering intelligence services by giving them new tools, access to unclassified data which can be shared with the public and with allies. It is also easing their load: cyber-security firms have been as important as Western spy agencies in defending Ukraine from cyber-attacks. But the boom is also challenging those services by blurring the line between the open and the secret, raising questions over what must be done by spies and what can be done by others. And as data become more abundant, more revealing and more central to geopolitical competition, questions arise about law, ethics and privacy. “The separation of private- and public-sector interests is a uniquely Western construct,” argues Duyane Norman, a former cia officer, “one that has great advantages but also important consequences.”

At the heart of this revolution is the internet. “We could not build our company if the internet had not become this core sensor,” says Christopher Ahlberg who leads Recorded Future, another firm which tracks bad actors online. “Everything eventually ends up on the internet.” The anonymity of the deep web, not indexed by search engines, and the dark web, which requires specialised software to access, makes them great places for terrorists, paedophiles and criminals. But that anonymity is often superficial.

Ghost in the machine
Flashpoint began life by using fake personas—imagine an analyst masquerading as a would-be jihadist—to burrow into extremist groups online and gather information about their intentions. It still does that. But its main business is now data. For instance, it tracks the “wallets” where extremist groups store bitcoin and other cryptocurrencies, says Mr Borene. The movement of funds in and out of such wallets can hint at impending terrorist attacks. Similarly Primerai, a firm based in San Francisco, was able to provide a government with eight hours’ warning of a cyberthreat by identifying hackers’ boasts on the deep web prior to an attack. The firm used natural-language processing, a type of ai, to analyse large amounts of text alongside the client’s proprietary data.

Another part of the industry illustrates a different model: rather than observing threats unfold on the internet, it monitors them from inside the network. The firms that build key hardware or software—think Google’s dominance in email, Microsoft in operating systems and Amazon in cloud computing—enjoy unparalleled insight into the private traffic that crosses their network. The result is a huge private machinery of signals intelligence, insights from which can be sold to clients as a defensive service. Microsoft tracks 78 trillion “signals” a day (such as connections between a phone and a cloud server), says Sherrod DeGrippo, the firm’s director of threat-intelligence strategy.

Analysts look for anomalies in those data and keep tabs on the tools, infrastructure and activities of established state or criminal hacking groups, known as advanced persistent threats (apts). Last year Microsoft revealed that “Volt Typhoon”, a Chinese hacking group, had targeted American infrastructure, including water and energy. More recently Mandiant, an information-security firm, has shown how the gru, Russia’s military-intelligence agency, was tied to the disruption of water utilities in America and Poland. These firms’ presence inside Ukrainian networks means that they see many threats that Western governments do not, says an insider.

Western intelligence agencies may break their countries’ laws under some conditions, for example by bribing foreign officials. Private firms cannot. But they have other advantages. “We can connect threat actors, their infrastructure and their targets in a way that intelligence agencies may not be able to,” says Mr Ahlberg. He points to agencies that may be authorised by law to operate freely abroad but not at home. “We can look abroad and look inside, in a way that is unique.”

Many firms jealously guard their data and are secretive about their methods and clients. They can also be surprisingly collaborative. Companies are watching the same groups of Chinese, Russian, North Korean and Iranian hackers. “We’re all going against the same threat actors,” says Ms DeGrippo, “but we all have a different set of data...The only way to get the complete picture is for us to work together.” Analysts also move between sigint agencies and threat-intelligence firms, bringing knowledge with them. Lewis Sage-Passant, the global head of intelligence at a large pharmaceutical company in Europe, says that in-house corporate intelligence teams can be similarly collegial. “They are cut-throat rival companies, but chances are the intelligence teams are…the very best of friends and they’re talking on an almost daily basis.”

“The uk intelligence community is facing an existential challenge”
For spies, much of this is good news. There is much they would like to spy on but, for want of time, money or other resources, do not. With satellites, the private sector has solved that problem. The growth of the commercial satellite industry allows states near-blanket coverage. In the past Britain bought hundreds of thousands of dollars’ worth of commercial satellite images each year, says Sir Jim Hockenhull, a British general. “Now we’re in the multiple millions.”

Then there are situations where the spies know a secret but cannot share it with allies or the public, for fear of revealing something about the source. This was common in space intelligence, says Aaron Bateman of George Washington University. During the cold war America rarely shared satellite images even with its nato allies, except Britain. Today, commercial satellite images are shared routinely. Governments can also tip off outside analysts to look for particular things; and those analysts stumble on intriguing things themselves. The result is “a larger workforce that the us government doesn’t have to pay for, but still benefits from”, notes Mr Bateman.

Not goin’ there, dude
That raises a wider question of life inside and outside the tent. Joe Morrison of Umbra, a radar-satellite startup, recalls being asked by Western officials why they ought to work with commercial unclassified vendors. “I said: access to talent that likes to smoke weed.” He was not joking. Intelligence agencies offer recruits the allure of working for organisations with sparkling histories and a patriotic mission. But waiting a year for security clearance, taking a pay cut and not being able to work remotely can be deal-breakers.

The most radical view is that Western intelligence needs to start from scratch. “The uk intelligence community...is facing an existential challenge,” argued Lucy Mason, a former British defence official, and Jason “M”, a serving intelligence official, in a paper published by the Alan Turing Institute in November. “It is being out-competed by providers of open-source intelligence and data companies.” Their solution was a new model, not one where national security is “done only by some cleared people in highly centralised, closed organisations”.

There is no doubt that non-secret sources are increasingly important. “If I’d gone and collected all of China’s military procurement records, I’d probably have got an obe,” says a former British intelligence officer, referring to a national honour. “The fact that they were, for many years, just sat there in open source completely bypassed everybody.” Location data scraped from mobile apps and traded by advertising brokers is now routinely used by intelligence agencies.

None of this means that those agencies can be abolished. The fact that public data can answer many questions that would once have required secret intelligence does not mean they can answer all such questions. Open sources shone a light on Russia’s pre-war military build-up in 2021. Nonetheless, only states had access to the most incriminating evidence, such as intercepts of Russian war plans and indicators that Russia was moving blood plasma to the front lines.

The second problem is that the value of public data often lies in fusing it with something secret. But crossing between unclassified (the “low side”, in the jargon) and the classified (“high side”) world is harder than it seems. An agency might want to compare publicly available records with secret intelligence on particular Russian intelligence officers. “What’s actually sensitive is the question you ask,” says a person familiar with these efforts. “As soon as the question comes from the high side down onto the low, that question, and the data you pull, is detectable.” Pulling troves of public data to the high side is too expensive—secret compute is a scarce resource.

The third issue is to do with the legal and ethical problems that arise in a data war. For Chinese intelligence services, a core part of their strategic competition with the West is about data. Over the past decade they have plundered huge data sets—government personnel records from America, electoral data in Britain, immigration data from India, phone logs from South Korea and road-mapping data from Taiwan.

Much of that is traditional intelligence gathering. Some of it is to enable China to catch Western spies. But it also has a larger and more sinister purpose. “Building databases of society has been [Chinese] intelligence …methodology since the 1930s,” writes Peter Mattis, a China expert and former cia analyst. “Start with the broadest possible data on individuals, then filter and target them for intel and influence.”

Western agencies are far more constrained. British spies can and do collect bulk personal data (bpd) from abroad. But if they want to retain or examine it, they need a warrant and must show that acquiring, keeping and using the data is proportionate to a specific aim. Hoarding it in case it is handy later will not do. It is easier for firms to collect bpd than agencies. The same is true in America. Emily Harding, a former cia analyst, says it is “hard or impossible” to “identify and scrub” data on Americans from large data sets—a legal requirement. So American agencies are “far behind private-sector entities with no such restrictions”.

The growing reach of the private sector also raises issues. In the 19th century the Pinkertons were used to infiltrate and intimidate unions. Today states, companies and wealthy individuals use private firms to spy on dissidents and journalists. Some worry that it is possible for less scrupulous governments to use their corporate counterparts to collect information or do things they themselves, lawfully, could not.

So far, these issues have played out among a small group of securocrats, lawyers and privacy advocates, because the public has not grasped how much of their lives is now recorded, tabulated, collected and traded. “Today, in a way that far fewer Americans seem to understand, and even fewer of them can avoid,” concluded a report for America’s director of national intelligence in 2022, commercial data “includes information on almost everyone that is of a type and level of sensitivity that historically could have been obtained, if at all, only through targeted…collection”.

This is the end
This report has shown that spying has become more difficult in many ways, and easier in some. Intelligence agencies will need to work harder just to keep up with the accelerating pace of technological change, from pervasive surveillance to ubiquitous encryption. The digitisation of the world, the data deluge and ai are threats to much traditional spywork. But those trends also create opportunities: from exposing the digital tracks of state hackers to democratising intelligence.

That is a potential boon to intelligence agencies, increasing the scale at which they can collect, widening what they can collect on and expanding the ways in which they can publicise things that might once have been too secret to share. Yet these technologies, more than the spy planes and satellites of the 20th century, are bound up with those of the civilian world and impinge on it more deeply. In the democratic world, at least, the agencies that fail to take the public with them will find that either their capabilities or their legitimacy will fall dangerously behind. ■

This article appeared in the Technology Quarterly section of the print edition under the headline “Intelligence, Inc”

Be careful out there!

Stay "Far from the madding crowd."

Amo-a,

-JackW