Fatfingers Create Kerfuffle at New York Times

Twitter lit up earlier today with news that The New York Times subscriber list must have been hacked. It seems a few million folks received messages purporting to be from the Grey Lady, advising that their subscription had been cancelled per the recipients’ instruction, and asking them to reconsider.

News that the message had originated from Epsilon Interactive, who earlier this year were themselves a target of a now-infamous ESP breach seemed to confirm the assumption that hackers had sent the message.

I’m a current subscriber and received my own copy of the message, so I had an opportunity to inspect the headers. The message seems to authenticate correctly; SPF designates the sending IP (which belongs to Epsilon) as a permitted sender on behalf of email.newyorktimes.com. The DKIM signature seems to have some formatting issues, and Gmail renders a “neutral” opinion on its authenticity

I think it’s safe to conclude that the mail did indeed come from Epsilon; the question is whether NYT’s account at Epsilon had been breached, or if the message was sent in error by an authorized user of the Epsilon account.

Word comes now from NYT that it’s the latter case – a NYT employee sent the message to over 8-million recipients in error; it was intended for only about 800 recipients. I’m guessing that the employee ticked the wrong box in Epsilon’s customer application, and selected one or more incorrect segments of their lists to receive the message.

The mistake is easy enough to understand and forgive, but it has to have been an awfully expensive one nonetheless. Recipients already on edge following the well publicized breaches were quick to assume the worst, and quicker to share those assumptions on Twitter. I am sure a significant number of recipients marked the message as spam, which will likely have a measurable impact on sender reputation, thereby hampering deliverability of future sends. Also, sending eight million messages is a lot more expensive than sending 800. The Times also sent a follow-up notification to recipients selected in error, essentially doubling the cost of the initial mistake. And it appears that the Times’ inbound call center was swamped with inquiries, which itself carries measurable cost.

There are probably a few lessons to be drawn from the incident. The one that springs quickest to my mind is, “Aim carefully.”

Edit: Headers included below, for the edification of various interested parties:

<pre>Delivered-To: andrew.barrett@gmail.com
Received: by 10.204.68.75 with SMTP id u11cs308277bki;
        Wed, 28 Dec 2011 10:14:50 -0800 (PST)
Received: by 10.50.17.195 with SMTP id q3mr36902675igd.11.1325096088086;
        Wed, 28 Dec 2011 10:14:48 -0800 (PST)
Return-Path: <150b39d46layfovciab7saeiaaaaaaxwmvxqqoseiuiyaaaaa@email.newyorktimes.com>
Received: from dmx1.bfi0.com (dmailer0121.dmx1.bfi0.com. [208.70.142.121])
        by mx.google.com with ESMTP id en3si23815262igc.11.2011.12.28.10.14.47;
        Wed, 28 Dec 2011 10:14:48 -0800 (PST)
Received-SPF: pass (google.com: domain of 150b39d46layfovciab7saeiaaaaaaxwmvxqqoseiuiyaaaaa@email.newyorktimes.com designates 208.70.142.121 as permitted sender) client-ip=208.70.142.121;
Authentication-Results: mx.google.com; spf=pass (google.com: domain of 150b39d46layfovciab7saeiaaaaaaxwmvxqqoseiuiyaaaaa@email.newyorktimes.com designates 208.70.142.121 as permitted sender) smtp.mail=150b39d46layfovciab7saeiaaaaaaxwmvxqqoseiuiyaaaaa@email.newyorktimes.com; dkim=neutral (bad format) header.i=@email.newyorktimes.com
Return-Path: <150b39d46layfovciab7saeiaaaaaaxwmvxqqoseiuiyaaaaa@email.newyorktimes.com>
DKIM-Signature: v=1; a=rsa-sha1; d=email.newyorktimes.com; s=ei; c=simple/simple;
	q=dns/txt; i=@email.newyorktimes.com; t=1325096067;
	h=From:Subject:Date:To:MIME-Version:Content-Type;
	bh=eAJBhggz56bI1iAGtnD6v787ib8=;
	b=XtriQSLHzmhMsaITYZDGYIS3VRsGlWGjP/3aELRkLaOawj6tlMWioBwo5yok6ipT
	rK73yfllp2Mk/NjAw4VBEOJtaRSwwhmGQOQKRp7rhi4aVtqXq5N8OJAExKKiH7pd
	GOJHgOIlmc42UkaqzlyQwJ/Zdppkp+coxwtB+Rwyt0Q=;
DomainKey-Signature: q=dns; a=rsa-sha1; c=nofws;
	s=ei; d=email.newyorktimes.com;
	h=List-Unsubscribe:Received:Reply-To:Bounces_to:Message-ID:X-SS:X-BFI:Date:From:Subject:To:MIME-Version:Content-Type;
	b=aVU70NLC7DPVnsy+oTRKCWYM8JFro8ZQ4q1rx4bKKIISSiLhLuq4lMayMhnZrKJN
	DTBukp3y6+dCQIv7VZgu1tXJ5BkcwQXZAuhBV2QH1RjaHiucsKuPX470y8Ybc25E
	76S+SiSLDknfSKurAlEJcmAQZyrx6f1WUvfVNcy3gUc=
List-Unsubscribe: <mailto:xxxxxx@email.newyorktimes.com?subject=unsubscribe>
Received: from [10.150.20.107] ([10.150.20.107:56117] helo=dlspvhcimailer7)
	by dmx1.bfi0.com (envelope-from <150b39d46layfovciab7saeiaaaaaaxwmvxqqoseiuiyaaaaa@email.newyorktimes.com>)
	(ecelerity 2.2.2.45 r(34222M)) with ESMTP
	id DD/83-28890-38C5BFE4; Wed, 28 Dec 2011 13:14:27 -0500
Reply-To: =?iso-8859-1?B?Im5vLXJlcGx5Ig==?= <150b39d46layfovciab7saeiaaaaaaxwmvxqqoseiuiyaaaaa@email.newyorktimes.com>
Bounces_to: nytimes.150b39d46layfovciab7saeiaaaaaaxwmvxqqoseiuiyaaaaa@email.newyorktimes.com

Holiday Fail, Indeed.

You might have thought that a large, e-commerce-centric company like Pro Commerce, Inc. – owners of well-know brands like ProFlowers and Red Envelope – would know better than to send e-mail with deceptive subject lines, a clear and blatant violation of the CAN SPAM Act.

You’d have thought wrong.

I received the solicitation below with the subject line, “Flower Delivery Notice Failure.” I immediately assumed the message was a phish, possibly sent with data obtained in one of the recent ESP breaches. After all, I do have a Pro Flowers account, but I haven’t purchased from them in a few months.

But the message is signed with a valid DKIM signature and is authenticated with SPF, and both point at network assets under Pro Commerce’s control. They are not using an ESP; the mail came from their own servers, hosted in Cogent IP space.

Maybe they should consider using an ESP. They obviously could use a little help. I realize that businesses of every size are under enormous pressure to make their Q4 revenue numbers, but this is not the way to do it.

Image

Peapod Fail

Nice checkbox.

It never ceases to amaze me how badly even big companies can get it wrong.