Update, 2:10 p.m.: In a memo sent out this afternoon, managing editor John Temple writes that the production problems were caused by a lost connection between the Post and its data center. The difficulties began at 10:30 p.m. last night and weren’t resolved until around 1:30 a.m., according to Temple.

“This meant that we were able to update about 40 percent of our metro delivery and single copy papers with the lead piece out of the Democratic convention, but we weren’t able to put the Nationals’ game in any papers,” Temple writes in the email, which can be read in full after the jump.

—

Bad luck for the Post‘s IT team last night, as their tech system was hit by perfect storm of trouble: The paper’s website was down, as was its Lotus Notes software, and its Methode writing software.

Problems abounded as a result. Post reporter Karen Tumulty found that her front page story was attributed to a Karen Tumlty, and Web problems hampered coverage of Michelle Obama‘s convention speech. Obama didn’t make the front page of at least some editions of the Post, prompting readers to complain to ombudsman Patrick Pexton that the paper was favoring Ann Romney when it put her on the cover during the Republican Convention.

It’s not clear when the tech mishaps started, but judging by an internal memo, they were resolved by 2:40 a.m. The Post was still able to produce papers in its Charlotte and D.C. presses, according to spokeswoman Kris Coratti.

“We experienced serious technical problems last night,” Post managing editor John Temple wrote to his staff in a morning email composed on his iPad, which was apparently working. “This meant that some readers received the regional edition and that we were unable to complete updating of website.”

Temple’s email thread turned into a rare reply-all spat for Post employees who were unhappy about the mishaps. One of the paper’s book critics, Jonathan Yardley, and environment reporter Darryl Fears got into a back-and-forth about how the paper should run a correction. Pexton chimed in that if the paper wasn’t going to run an explanation, he wanted an interview with Temple.

“This is no moment in the history of [the Washington Post] to let readers ‘fill the void with their own reasons,’ none of which is likely to bring in new home-delivery subscribers,” wrote Yardley.
John Temple’s afternoon memo about the tech problems

As you all know by now, we experienced technical issues last night involving The Post’s computer network that caused the newsroom to lose connection with our data center and limited our ability to publish both to print and the web. The problems began at about 10:30 p.m. — after we had published the first edition — and lasted about three hours. This meant that we were able to update about 40 percent of our metro delivery and single copy papers with the lead piece out of the Democratic convention, but we weren’t able to put the Nationals’ game in any papers. We did publish and deliver a strong paper to all of our readers, as well as a fully-updated special convention edition in Charlotte. Our publishing limitations also meant that on the website we had limited ability to update stories for several hours.

We had a hard-working and fast-thinking team on hand last night who worked into the early morning trying to trouble shoot the technical problems and devise work-arounds to enable us to publish outside normal channels. Through the cooperation of the tech team, circulation and production we were able to sharply limit the disruption to readers.

We’ve completed a post mortem of the technical issues and believe that we have a plan to address them.

Marcus, Liz, John, Shailesh