Data-Mining

by emptywheel

I've been arguing for two years that the secret that Bush was hiding about the illegal domestic wiretap program is that they were using crappy data mining programs to pick their targets for wiretaps. In tomorrow's NYT, they're almost done filling out that picture.

A 2004 dispute over the National Security Agency’s secret surveillance program that led top Justice Department officials to threaten resignation involved computer searches through massive electronic databases, according to current and former officials briefed on the program.

Though Shane and Johnston don't seem to have a sense of how using data-mining to pick targets might introduce problems into the entire concept of probable cause.

It is not known precisely why searching the databases, or data mining, raised such a furious legal debate. But such databases contain records of the phone calls and e-mail messages of millions of Americans, and their examination by the government would raise privacy issues.

Somewhere--I'll have to look for it--there's a reference to the program expanding once they got Abu Zubaydah's computer. Which suggests they took a data profile of one terrorist and all the contacts he had in his laptop, and assumed that everyone else with a similar profile was also a terrorist. I've long suggested that they may have wrapped up folks who shopped the same falafel joints as genuine Al Qaeda targets (ut oh--Bill O'Reilly may be in trouble).

Though there are signs this is a limited hang-out of the program. I suspect, for example, it may focus on an aspect of the program  that was ended--the bad data mining, while ignoring an aspect (US to US communication) that perhaps wasn't ended.

Some of the officials said the 2004 dispute involved other issues in addition to the data mining, but would not provide details.

And it wouldn't be an NYT article if it didn't include self-serving justifications. Why is it, do you suppose, that the NYT has waited until today to tell us they were conned into spiking their story in October 2004, just weeks before the presidential elections, because the Administration claimed there had been no disagreement on the program???

The first known assertion by administration officials that there had been no serious disagreement within the government about the legality of the N.S.A. program came in talks with New York Times editors in 2004. In an effort to persuade the editors not to disclose the eavesdropping program, senior officials repeatedly cited the lack of dissent as evidence of the program’s lawfulness.

[snip]

Mr. Gonzales’s 2006 testimony went unchallenged publicly until May of this year, when James B. Comey, the former deputy attorney general, described the March 2004 confrontation to the Senate Judiciary Committee.

I mean, you'd think an entity dedicated to the art of journalism might, when they heard Gonzales' claim in February 2006 that there had been no serious dispute ... you might think they would have revealed that they had been sold that bill of goods two years earlier, only to learn those were fraudulent goods. But I guess NYT thought they'd get around to telling us this in their own sweet time, huh?

Comments

[cboldt]

-- Though Shane and Johnston don't seem to have a sense of how using data-mining to pick targets might introduce problems into the entire concept of probable cause. --

It's a knotty balance.

With greater surveillance, ostensibly aimed at gathering foreign intelligence, what to do about overhearing a purely domestic hit contract?

Pardon the stream of consciousness hypothetical situation.

I've resisted forming an ultimate opinion on both the data-mining and (so-called terrorist) surveillance programs. I hold that "against FISA equals unconstitutional" is false, so you see generally where my head is.

Some issues are resolved on a macro level, and others on a fact-specific "after being charged" level.

My concern was, I'm sure, seen by others before I did, and I've seen it stated (albeit rarely). But the combination of "we think this person is bad" and "we have the right to incarcerate base on our belief" is dangerous.

The trade off with more aggressive surveillance is a risk of prosecution lost for violation of privacy. It's not an easy call, and a thoughtful person will resist jumping to a conclusion in either direction.

[bmaz]

You read my mind. After reading masaccio's link to this article, I was going to post a comment that is basically a nutshell of your post here. 2006, my butt; what do think they are sitting on or "spiking" as we live and breathe this moment. simply amazing, and yet they continuously fail to be students of their own failed history. Secondly, and this goes back to something I said earlier, NO WAY it is intellectually or legally viable to say this wasn't the same program, as Gonzales has done. Fraudulently hiding hiding aspects of their illegality does not detract from the fact that it is, well, illegal. Third, someone earlier pondered whether or not this junk could still be going on today; of course it could, and of course it is. Lastly, contrary to the bleatings of the Administration and their sycophants, this is not a complex and/or complicated legal area. It is fucking crystal clear, and in essence is no different from the analysis of pen register devices that has been going on for decades.

[AmIDreaming]

Suppose, now, just suppose you've just been effectively blindsided by two jetliners flown into skyscrapers. Suppose further that your shiny new SIGINT toy TRAILBLAZER isn't worth its weight in scrap metal, and your bosses are screaming for intel.

What choice do you have but to drop completely any USSID-18 protections and open the floodgates to any and all collection? Even though USSID-18 violations have heretofore been among the most serious transgressions which a SIGINT spook could commit...

[bmaz]

cboldt - I see exactly where you are going with that stream of consciousness; and on this point, I am going to diametrically diverge. Irrespective of our two positions, my overaching concern is, not just the failure to attempt to process all this within the framework of the law, but the absolute refusal to attempt it coupled with the dishonesty and disingenuousness attendant to that refusal. It is my inclination that you will agree with that part, but I am interested to hear.

[Garrett]

crappy data mining programs

You got your training set. Then you got your test set.

If the system has never passed a test set, no one on Wall Street would ever bet a penny on it.

Good-enough-for-government terrorist data mining program: first name, "Mohamad"; last name, "Atta". Hey, it works!

[lizard]

I am still fairly certain that the issue Comey et. al. objected to had absolutely nothing to do with the infringement of the rights of individuals. Rather it was the use of governmental authority against BUISINESS INTERESTS. Comey, Ashcroft, Mueller, were acting in character and defending what they always have defended, the rights of the wealthy. Specifically, I think it was the telecomm people who enabled the data mining program that were starting to push back against the NSL's. Maybe they foresaw the lawsuits they are facing now.

[bmaz]

AmIDreaming - the choice is to make a good faith attempt to bring it all within the law; even if it is after the fact and after the immediate panic has subsided. At a minimum, that is the choice.

[AmIDreaming]

bmaz - Of course. But Michael Hayden, DIRNSA at the time, has publicly shown himself to be pretty vague about probable cause.

[cboldt]

On a review, I find my comment too cryptic.

First, a restatement of one comment: But the unchecked, unilaterally-decided combination of "we think this person is bad" and "we have the right to incarcerate base on our belief" is dangerous. Our system of government, and our system of justice are based on checks and balances, independent prying (and sometime adversarial) eyes.

My other thought is more difficult to convey, but I tried with "The trade off with more aggressive surveillance is a risk of prosecution lost for violation of privacy."

A person who promises to prevent crime (or prevent terrorist acts - the line between "crime" and "terrorism" is is a tough draw - what was the crime before 9/11 was perpetrated?) ... at any rate, a promise to prevent crime is an interesting notion in its own right.

The protector (who has no liability on failure to protect) urges relaxation of privacy in exchange for exposing more potential criminals. But if the surveillance is unreasonable under the 4th amendment, then the prosecution is stymied.

The point of my post isn't to defend one conclusion over the other - I honestly find this to be a sticky and difficult problem. It should be an apolitical problem. I view it as tension between the hive and the bee.

[bmaz]

Damn cboldt, and you thought the FIRST comment was too cryptic?

[earlofhuntingdon]

A little knowledge is a dangerous thing. Why do seemingly more and more journalists know or tell us less and less about the context of their stories? "Might" raise privacy issues? Duh.

Data mining - the analysis done after data has been collected - may well be the tip of the iceberg, which iself doesn't seem to have broken the surface. I suspect that "commercial databases" are only one of many sources of data that newer and more sophisticated s/w is combing.

Extensive privacy issues are at stake. There are general classes of potential abuse that should be investigated. In addition, I suspect that few regular readers of blogs such as this would trust Karl Rove to keep his hands off data flows that would let him either improve his "math", or obtain J. Edgar Hoover-like blackmail files for current or future use.

[cboldt]

-- Irrespective of our two positions, my overarching concern is, not just the failure to attempt to process all this within the framework of the law, but the absolute refusal to attempt it coupled with the dishonesty and disingenuousness attendant to that refusal. --

First, the hypothetical was for an investigator to choose between exposing a domestic hit, and exposing surveillance outside of constitutional bounds.

I think there is a big area between FISA and "foreign intelligence gathering for which no court need be involved," but I see no way to distinguish between the two without an actual case.

IOW, I'd not be keen on arguing that all surveillance outside of FISA was unconstitutional. My reaction to "it's illegal" (i.e., outside of FISA) is "So what? - tell me why it's unconstitutional."

I think you and I are similarly offended by the amount, extent and nature of official prevarication.

There are trade offs involved, and prevaricators dismiss the trade-offs by invoking bogey-men.

At any rate, "privacy" and "individualism" are extraordinarily fundamental and difficult notions.

[cboldt]

Not necessarily a cure for cryptic, but here goes.

The more that privacy is violate, the more likely the looker will find a pre-crime.

But if the privacy violation is found inconstitional, then the pre-criminal goes free, under the same protection that got him off the first time.

How much personal privacy will ALL of us give up, to protect the hive?

[bmaz]

That I will agree with.

[jazz]

Maybe I am just blowing in the wind, but once more with all that AL and EW and the rest of you have figured it is time to file a bar complaint. I do not see anyone in Congress ready to do the heavy lifting required for impeachment. Just maybe get the stone rolling down the hill with a bar complaint against Gonzo for lying will wakeup some bravery and put some guts into our Cogressmen.

[QuickSilver]

Well, Jay Rockefeller wasn't cryptic. He said in his letter to Cheney it sounded like TIA.

From the size of government contracts for data storage, it sure seems like they are saving every damn thing.

[the third man]

bmaz, re: cbolt
hear hear!

Wouldn't it be great if on the 33rd anniversary of this letter there was a similar one from George and Dick? But please hurry, just 12 more days!

[bmaz]

Just to clarify, my agreement was with the 18:44 comment; I had not yet seen the 18:49 comment when I posted. As to the 18:49 comment, I am not sure whether you have failed to cure the crypticness, whether I am dense, both, or neither. I trust you meant to type unconstitutional; if not, then the dense explanation is looking better. At any rate, and I fully admit that reasonable people may differ, but, to me, "the hive" IS privacy, our other rights and the constitution. I think any of our lives individually is subservient to the principles and the process. That is one thing that eats at me at the Administration's line of bologna about fighting them there to protect us and it was necessary to incur the losses over there to protect us. We have lost somewhere on the order of 3,700 soldiers in Iraq; close to 5,00 if you include the civilian contractors that have been lost (and which would have been included casualties in any other war). And for what? Most all rational people, by now, agree that Iraq has made us less safe overall. But, even if the converse were true, are you (figuratively, not cboldt or anyone else individually) telling me that 5,00 americans would have died here without going? No. This has been one of the biggest shams in the history of the world. As you have eloquently stated, the "prevarication" and "straw men" artifices that we have been plied with are, indeed, unconscionable.

[Boo Radley]

Thanks bmaz and cboldt for an excellent discussion. Another metaphor might be the tension in Xtian theology between social and individual responsibility. Another take is that privacy and the public good are "permanent existentials" that overlap in difficult cases.

[the third man]

letter
(sorry, screwed up the link)

[bmaz]

Twice I mistyped "5,00" when it should have been 5,000. Consistent if nothing else I suppose.

[cboldt]

Tied up in combination with the power to peer is the power to unilaterally incarcerate/impose on a pre-criminal without answering to a third party.

At any rate, there should be no doubt in anybodies mind that some pre-criminals aim to fly airplanes into buildings, or inflict similarly destructive events aiming to maximize loss of innocent life.

It's a difficult and high-stakes balance.

What sticks in my craw is that the observer (Bush/Gonzales) fails to describe the balance except in black and white, and that the talker (Gonzales) comes off as a liar.

I felt about the same with Clinton/Reno, FWIW - although the incidents were different.

Listening to GWB on his morning talk, he points to prepaid cell phones and the internet as the technologies that require FISA adjustment. I'd concede both of those technologies as the equivalent of billboards, if the government would promise that non-prepaid cell phones had privacy. Dream on, ALL cell phone conversations are the equivalent of billboard pronouncements.

Do read the FISA changes carefully, just as it was prudent to read interrogation methods changes carefully.

[cboldt]

-- but, to me, "the hive" IS privacy --

No, I mean for "the hive" to represent submission of individual, for the greater good. The individual is expendable, if it benefits the hive.

I'm "anti-hive" in one way - that humans don't hive. I think individual personal independence and responsibility creates a superior human society.

[om/]

Plame was using NSA for domestic political group survellance. The web sites make sense because some of the targets like MoveOn, Larry, etc. had Plame complain about domestic groups and CIA using NSA, etc. on these politics.

[marksb]

Of course they are capturing and saving *everything*. They can, so this gang will.

As we've discussed before, the technology exists, off the shelf, to split the optical signal, then capture and do RT top-level analysis of all backbone traffic. "Legal intercept" gives them access to the switches and that's been in the contracts with telcos since at least 2001. This is a project that, if given to someone who actually can do a decent engineering job, could be done in fairly short order.

To expect these guys to do anything else is wishful thinking.