Wednesday, April 04, 2007

Computergate: Let's play dodgeball! (UPDATES 1, 2 and 3)

Copmputergate -- or emailgate, or what-have-you -- has been quiescent these past few days. But now we learn more about the White House effort to bypass scrutiny by running emails through private RNC servers.

The Chattanooga Times Free Press has a story called "Morphing a conspiracy?", which reveals that the photo of Karl Rove carrying a Coptix brochure (Coptix being one of the Tennesee companies involved) was faked.
"It's easy for people to plant disinformation and misinformation out there," said Josiah Roe, executive vice president of Coptix, based in St. Elmo.

Mr. Roe said the company altered the photo and placed it on the Internet after bloggers implied that Coptix was involved in a "vast right-wing conspiracy" because the company -- along with another local firm, SmarTech -- provides an Internet service for the Republican National Committee.
Cute, Josiah. You've done your country a real service.

Now here's the juiciest part of the story:
Internet bloggers also have alleged that Chattanooga-based SmarTech is involved in the case of e-mails sent on a nongovernment account that relate to the firing of the eight U.S. attorneys.

Jeff Averbeck, president of SmarTech, said the company hosts a server for the RNC that translates Internet domain names into numerical addresses. He said Coptix also holds an additional copy of that information -- which he said is like a "telephone book" -- for the RNC.

Mr. Averbeck said his firm does more for the RNC but he declined to elaborate, citing client confidentiality. But he said that SmarTech does not have access to RNC e-mail records.

Mr. Roe, with Coptix, said his company also does not have access to that RNC information.

RNC spokeswoman Tracey Schmitt said the committee "is not a client of Coptix."
Perhaps one of our readers with a knowledge of how such companies operate can make something comprehensible of this statement. Here's how I see it:

The White House has released copies of emails in which officials doing (shady) government business use the GWB43.com email address instead of the .gov addresses which is stored as mandated by the Presidential Records Act. That address, GSB43, traces back to SmarTech and Coptix. SmarTech merged some years ago with a company run by a Bush family friend; the company handles everything Republican.

But SmarTech now says it has NO access to RNC email records.

And SmarTech makes this announcement just after Henry Waxman, Chairman of the House Oversight and Government Reform Committee, issues a formal demand to the Republican Party to turn over emails -- because either the Presidential Records Act (which mandates a historical record) or the Hatch Act (which forbids White House workers from doing party business on the taxpayer's dime) has been broken here.

No records here, sayeth SmarTech -- and SmarTech sayeth no more.

Looks to me like what we got here is one hell of a game of dodgeball.

UPDATE: So just how easy -- or difficult -- would it be to erase all record of those emails? Geek-speak translated into human-speak, after the jump...

(To read the rest, click "Permalink" below)

The best explanation comes from a tech guy named meldroc, who posted over on Democratic Underground (with paragraph breaks added for increased readability):
Let me clarify how much I know about computers. I am a software engineer. I do this for a living. I used to work for a major hard disk manufacturer, and I've also worked for a major computer manufacturer on two occasions, working with high end servers and mainframes. I know a lot of the technicalities about what happens when you delete stuff.

Traces would remain. To start with, deleting a file such as an email usually consists of removing the directory entry in the file system pointing to the file and marking the disk sectors where the file resides as free. The contents are still there, and frequently can be recovered by undelete utilities. If we're talking about a database or a mail server file, removing an email there will also leave traces behind.

Let's say you go all the way - format the disk. That will still leave data behind that can be recovered with forensics tools. Go a step further and wipe the entire hard disk. Now you're getting somewhere - you overwrite the entire hard disk. But... For one, it's now obvious to everyone with enough neurons to form a synapse what you've just done. What do you think Waxman and the rest of Congress are going to do if they hand you a subpoena and you hand them a hard disk containing nothing but random data? If you do that, you're going to jail.

Also, that server's not the only place where those emails are stored. You have routers on the Internet where those emails passed through - those may have logs showing that those emails existed. You have the copies of emails saved on individual people's workstations. There may be hundreds of workstations out there with copies of some of the emails.

In the DOJ document dumps we've already seen are references to some of the emails on gwb43.com - when those servers are subpoenaed, they'd better produce those emails if they know what's good for them. And those emails may reference other emails - better produce those too.

Then there are backup tapes. If we're talking about a halfway competent ISP here, there will be an entire system in place for doing backups. In a typical environment, you'll have seven tapes, one for each day of the week, used for differential backups (where you only back up stuff that's changed or added since the previous day's backups.) You'll have four tapes containing full backups of everything, one backup done each Sunday, so those four tapes will keep the state of the systems for the entire previous month. Then you'll have a permanent backup for each month that goes into storage.

In short, not only can you bring back the data on the email server as it was today, you can see what was on the machine yesterday, or last week, or six months ago, or three years ago.

On top of that, you'll have off-site backups - you'll make a backup onto a tape once a month, and that tape's stored elsewhere, like in a safety deposit box, just in case the building burns down. Depending on how badly the ISP wants to be able to put itself back together quickly in case of a disaster, there may be a couple dozen backup tapes containing those emails.

On top of that, we won't be talking about one server. There may be two, or even more servers, operating in parallel to ensure that email service is up and running at all times. And those servers won't have just one hard disk. They'll be running a RAID array with a whole bunch of hard disks (anything from four to dozens, depending on the scale), designed with redundancy in mind so if a hard disk, or even several hard disks fail, the data will still be there.

In short, if Waxman subpoenas the entire contents of gwb43.com's email system, saying "Oops, we lost a few messages." is NOT going to fly. Nobody who knows how these things work will believe that the email equivalent of Nixon's 18 1/2 minute gap can be produced unintentionally. Those emails are in that system, and if the RNC is following the law, they'll be able to produce them in a matter of minutes. If they say they can't they are LYING.
That said, I think there is still room for an "Oops" factor -- as in "Oops, we were a terribly incompetent ISP and we did not keep backups." Obviously, if someone tries to "oops" his way out of this mess, anyone with any brains will know that heavy-duty hugger-mugger has happened. But we still won't have the damn emails. And as long as the oops-er comes up with a story that a jury might find remotely plausible, then he can escape obstruction of justice charges.

Or so I presume.

And why do I make that presumption? Because right now, Jeff Averbeck seems to be acting awfully damn smug.

Oh -- and please read ViViDVeW's commentary below. We have a pressing need now for the full email headers. And if Averbeck says he can't get that...

UPDATE 2: Here's what ViViDVeW has to say...
While the DNS records for GWB43 may point to SMARTech servers this does not mean ANY emails reside there.

As I have stated before, their servers have all of the appearance of being only mail gateways. Think of them as mail relay agents. The technical term is MTA (Mail Transfer Agent).

When an email is sent to a gateway, it looks up the domain (the part after @) to see if it is one that they control. If so, it then simply forwards the email to the IP address listed for that domain which goes to another relay or its final stop, the MUA (Mail User Agent). The MUA is the place where all emails are stored. The clients download their email from the MUA. The MUA may not even handle outgoing mail for the clients – if so, even headers from an email sent from this domain may only show you the server that sends email, not the one that stores them.

I have checked the document dumps from the Judiciary Committee and none that I can find have email headers or even email addresses. They are all scans of printouts in a “presentable” format with none of the needed technical stuff showing.

This is really a load of crap.

A look at ANY email delivered to this domain or even the email settings for a client computer of this domain will tell you what server stores the emails.
Okay, so if SmarTech is just a the go-between, which company has physical possession of all those backup tapes?

Allow me to offer two suggestions:

1. A company owned by Bush family friend Mercer Reynolds. One of his tech firms merged with SmartTech. See the expanded version of my first piece on this scandal, here. (The expanded version was published on another site; the Cannonfire version did not discuss Reynolds.)

2. A company owned by Michael L. Connell; see here.

UPDATE 3: Those cocky guys at Coptix bray about their photo retouch work and show how it was done. They also say that the photo was reprinted on this site, which is not true (although I did provide a link to the Corrente site). The bug-eyed monster shot was a nifty touch, fellows. I know a thing or two about Photoshop myself, and now that I look at the piece more closely, the overly dark arm shadow does seem wrong. It is fitting, Josiah, that you chose a restaurant scene -- because, as you may have heard, certain dishes are best served cold.

Be seeing you!

17 comments:

ViViDVeW said...

>>Mr. Roe, with Coptix, said his company also does not have access to that RNC information.

This statement could be technically true and IF they are smart it is true. While the DNS records for GWB43 may point to SMARTech servers this does not mean ANY emails reside there.

As I have stated before, their servers have all of the appearance of being only mail gateways. Think of them as mail relay agents. The technical term is MTA (Mail Transfer Agent).

When an email is sent to a gateway, it looks up the domain (the part after @) to see if it is one that they control. If so, it then simply forwards the email to the IP address listed for that domain which goes to another relay or its final stop, the MUA (Mail User Agent). The MUA is the place where all emails are stored. The clients download their email from the MUA. The MUA may not even handle outgoing mail for the clients – if so, even headers from an email sent from this domain may only show you the server that sends email, not the one that stores them.

I have checked the document dumps from the Judiciary Committee and none that I can find have email headers or even email addresses. They are all scans of printouts in a “presentable” format with none of the needed technical stuff showing.

This is really a load of crap.

A look at ANY email delivered to this domain or even the email settings for a client computer of this domain will tell you what server stores the emails. Hell even without, if I were a reporter and had the legal protections they do, I could still probably find out this stuff in a few hours.

ViViDVeW said...

I agree with all the technical details of meldroc’s post, but I think he puts to much likelihood on the possibility that an ISP hosts the email servers. These are not like the consumer email accounts that people have i.e. joeshmo@isp.com

The RNC owns its own domain. This is more of an enterprise email setting.

If someone asks shadyinternet.com where there servers and backups are, the answers could be “in our data center in black hole country X and oh by the way – we are having problems with those servers and, no, we don’t keep backups”.

I’m not saying I have any empirical evidence to suggest that they have gone to these lengths. It could be how meldroc described it, how I described it, or anywhere in between.

Anonymous said...

there is nothing to stop these folks from saying something like, "we ran out of harddrive space so we junked the email servers we used in 2004/5 and didn't keep any backups, why should we since we don't want any email trail anyway and haven't been subpeonied yet?"

If they toss the harddrives away and smash them with a hammer, there is no way to recover any data and frankly I have heard of several companies doing just that: instead of donating their retired servers, they don't want any data to be recovered from the hard drives even if they haven been wiped and formatted, so just cheaper and safer to physically destroy them.

these folks will never give up any evidence that could incriminate them--why should they when they are in control anyway?

the worst that happens is that the judge slaps them on the hand for not saving their email records when they upgraded to a new email server.

Anonymous said...

maybe vividvew will be able to answer. But just to prove it is not a mail relay, but is in fact email servers. Here's a hint mail.smartechcorp.net

ViViDVeW said...

Wow mail.smartechcorp.net.
There really is just no way I shouldn’t have figured that out for myself.

YUP, that looks like the one. mail.smartechcorp.net (64.203.96.49) is running all the services expected on an MUA.

Pop3, which is a client access protocol, and a web server, that looks like it is for webmail access (just point your browser at mail.smartechcorp.net and have a look). It is also running SMTP so it’s a good bet that this is also the outgoing mail server for accounts in the email domains it serves.

Nmap is 93% sure that the OS is FreeBSD which would jive with the OS on the mailscan servers. I’m going to go with nmap’s guess unless someone is willing to do a more “intrusive” investigation. I doubt the OS matters at this point.

A quick check of the WHOIS for 64.203.96.49 says that it belongs to SMARTech.
The IP address (64.203.96.49) is also in the same block assigned to SMARTech (64.203.96.0/20) that is being used for the mail gateways (64.203.97.101 and 64.203.98.245)
http://www.dnsstuff.com/tools/whois.ch?ip=64.203.97.101
http://www.dnsstuff.com/tools/whois.ch?ip=64.203.96.49

While it is still technically possible that the emails themselves are on another server(s), network file systems or GRE/VPN/L2LTP tunnels for the geeks out there who care, I doubt this is the case. I believe this IP address leads to the server, or cluster of servers, that hold the email stores.

If my earlier assumption that SMARTech’s servers are located in the AirNet data center in Chattanooga is correct, then it would follow that this one is as well. Traceroute leads me along the same path to all three IP address and it just makes sense for performance reasons. Its now hard too see how SMARTech wouldn’t have access. It’s possible they don’t administrate the servers or have the passwords, but if told “give us the hard drives”, “we don’t have access” would be a bullshit answer.

It is now my opinion that there is a very good chance that Jeff Averbeck was lying on both a technical level and a plain English level.

P.S. Cannonfire is still on the first page of google hits for GWB43. A week ago google was only giving me a few pages of hits for GWB43, now there are over 100,000.

Joseph Cannon said...

ViViD, he may not be lying -- in a sense. I think we are dealing with a case where a single company is pretending to be different entities. That's what was going on with Connell, who set up an allegedly separate biz under his wife's name.

Anonymous said...

Geez VV, a little more literal. Not whois.. think google.. "search.term" (You did want email headers??)

And no for reference, New Media is the frontend web developer, SMarTech is the servers, Coptix (used to be) DNS and IP allocation. Three legs of one machine. (of course DNS is lame and contracted(?) out by smartech.

Anonymous said...

So they do run POP3.

Someone might be able to find something interesting in docs at archive.org, including their acceptable use policy:

They tell their clients not to send

"E-mails containing forged or falsified information in the header (including sender name and routing information), or any other
forged or falsified information."


Keep digging; we don't know what we'll find.

b

ViViDVeW said...

>>ViViD, he may not be lying -- in a sense. I think we are dealing with a case where a >>single company is pretending to be different entities. That's what was going on with >>Connell, who set up an allegedly separate biz under his wife's name.

I’m not sure I fully follow, as I think I’m getting lost in layers of obfuscation and bullshit.
It now seems that SMARTech controls more from a technical point of view that the hollow shell it seemed to me at first glance.

The last two “hops” before hitting the servers are
cha-core-01-edge.smartechcorp.net
cha-cust-01-core-01.smartechcorp.net

To me this means that they have more than a few servers rented in someone’s data center that they have little control over. The core/edge naming scheme implies a medium size network. Since they only seem to do business with RNC et al...

>>Jeff Averbeck, president of SmarTech, said the company hosts a server for the RNC >>that translates Internet domain names into numerical addresses. He said Coptix also >>holds an additional copy of that information -- which he said is like a "telephone book" >>-- for the RNC.

He made it sound like they just do a yellow pages lookup and send the emails on their way, but he did admit that SMARTech is the host of the telephone book server (email gateway). In this context “host” can only mean own, run or both. So if they own or run the gateway server, then if follows that they own or run the one that has the emails on it too, which I’m like 99% sure is sitting in the same room as the one he admits they host.

I think SMARTech either owns or administrates the servers or both. I think he’s full of crap.

>>Geez VV, a little more literal. Not whois.. think google.. "search.term" (You did want >>email headers??)

Not sure what you mean here but the reason for wanting the email header was only to find the MUA which it turns out is at the obvious address mail.smartechcorp.net. I’m not sure what else a header would tell me at this point. If the headers were false then I suppose that would tell you but I have no reason to think they would be.

Anonymous said...

this is completely off the techno topic (waaay outa my league there), but gotta wonder if coptix et al., (read 'rove') put the photo out there, you know, along the lines of the dan rather spike.

oh you libruhl conspiracy nuts, you'll do anything, fall for anything, therefore nothing you say has any credibility.

now, what were we talking about? oh right, let's stay on topic...the pleasure of the president?.....

ViViDVeW said...

>>Jeff Averbeck, president of SmarTech, said the company hosts a server for the RNC
>>that translates Internet domain names into numerical addresses. He said Coptix also
>>holds an additional copy of that information -- which he said is like a "telephone book"
>>-- for the RNC.

Re-reading this I think I have been mistaken about what server he was talking about. “telephone book” could refer to the DNS server that SMARTech runs for the gwb43.com domain not the mail gateway that does “email domain” lookups.

In geek, the MX record lookup not the email forwarding domain lookup.(wow, its so much easier to talk about these things in geek instead of English)

But EITHER way, I justed wanted to clarify a possible error on my part.

The DNS server is also on the same network and in the same place I believe all the others to be, so he’s still full of crap.

Anonymous said...

geez you really don't know how to use google. anyways, their is mailscan1. and mx1. (assuming mailscan is for client who get spam filtered)

The mx2. and mailscan2. are just redundant and on slightly different network (core / cust / bell south/etc)

The nameservers used to all be ns.trespassers-w but now are all half/half or all ns1./ns2.smartechcorp

FYI, tresspassers-w is a offsite backup for DNS in Rackspace NOC.
Currently, I think they have two OC-3 serving their basement AirNet NOC and two OC-12 serving downtown.

Anonymous said...

Joe, have you sent this post and thread to Henry Waxman? (BTW, I lurk here daily but rarely post)

Joseph Cannon said...

frankly, Laura, I'm not sure how to go about that, or who in Waxman's office I should approach...

Anonymous said...

Received: headers from the email as delivered to the recipient should tell you all servers it's been through. Who knows, forgery may have occurred... They will tell you the sender's real IP address...

Message-ID: - if it exists - added by originating MTA - so...?

Return-Path: - supposed to give the envelope sender's address.

Kudos to you, vvw, but I wouldn't hold with the idea that full headers are a red herring because we've no reason to think they might tell us anything we don't already know! Odds are they will tell us a lot.

They were stupid enough to say in writing, let's do this stuff to avoid the possibility of legal attention. Who knows what else they are or aren't stupid enough to do?

om

Anonymous said...

You can find the name, position, & email of everyone on Henry Waxman's staff by going to the web site www.outsourcecongress.org where someone has written a utility that looks up every staffers name! Waxman's staffers, from that site are: Kim.Alfred@mail.house.gov,
Becky.Claster@mail.house.gov,
Patricia.Delgado@mail.house.gov,
Greg.Dotson@mail.house.gov,
Zahava.Goldman@mail.house.gov,
Karen.Lightfoot@mail.house.gov,
Amanda.Molson@mail.house.gov,
Karen.Nelson@mail.house.gov,
Phil.Schiliro@mail.house.gov,
Rachel.Sher@mail.house.gov

Simple huh? I'm going to stick my neck out & assume Henry's email is Henry.Waxman@mail.house.gov Just a hunch.

It looks like Phil Schiliro is his Chief of Staff...but heck...send the info to all his staff! Do you really think the person assigned to firearms is busy with that topic this week? Probably not. Just get what you have to everyone!

Btw, SmarTech seems to have two colocated sites...they claim "Class A" operation & imply that they ave hot site failover/backup including electrical power. They're lying if they say they don't have copies of the emails. Looks like each email might even be traceable to PC Workstations by IP Address if the user chose to use the SQWebMail feature to authenticate via IP address. GO GET 'EM!

Anonymous said...

By the way, you are missing another company that seems to be involved or might know something, at least according to someone at Coptix:

http://episode49.com/portfolio/National-Republican-Senatorial-Committee.html