Robots could do better than some journalists

When the Associated Press announced plans to use computers to write corporate earnings stories, a number of journalists asked me if I was as horrified by the prospect as they were.  In fact, I think robots could do better than some reporters.

With all respect and affection for my fellow journalists, I have concluded that a well-programmed set of algorithms can be far more analytic and precise than the sorts of harried, math-averse humans who are widely employed to write about complex business matters. Here is a case in point:

Poynter.Org led a story today about Gannett’s second-quarter earnings with a headline saying “Circulation Revenue Rises at Gannett Local Papers.” The problem with the headline is that the press release and the accompanying financial tables provided by the company showed unambiguously that circulation revenues actually FELL in both the first and second quarters of the year.

For the record, as reported in the press release, circulation revenue at Gannett’s newspapers for the first half of the year was down 1%, ad sales revenue was down 5.3% and “all other publishing” revenue was down 2.4%. 

So, how did the Poynter reporter get it wrong?  Because, in his haste to crank out a story, the author evidently relied on the bafflegab in Gannett’s press release, instead of looking at the several pages of detailed financial tables appended to it. In fairness the writer, who was alerted to this issue but so far has not amended his article, what human wouldn’t be confused by the following statement from the company:

“Circulation revenues were $277.9 million, down just 0.6 percent from $279.7 million in the second quarter in 2013. An increase in circulation revenue at Newsquest [GCI’s division in the United Kingdom] was offset by circulation revenue declines at domestic publishing operations. At local domestic publishing sites, home delivery circulation revenue was up in the quarter due, in part, to strategic pricing actions associated with enhanced content.”

You can’t blame Gannett for trying to put the best face on the umpteenth weak quarter in a row for its publishing operation. And you can sort of see, sort of, how a time-constrained journalist fell into the PR trap by seizing his lede from a fragment of the third sentence in the sixteenth paragraph of the press release. But a well-programmed computer could have done better.  

A half-decent, natural-language engine could have assimilated, organized and analyzed the facts and figures provided by the company in far less time that an ordinary human could read, much less unpack the meaning of, the document.

The robot would organize the data into normalized tables for instant publication and then drop the key information into templates designed to produce concise and understandable narratives. Knowing in advance the Wall Street consensus on a company's upcoming earnings, the robot could determine whether the company beat or failed to meet investor expectations. Templates would be pre-programmed with dictionaries that would know a drop in revenue from 2013 to 2014 was, depending on the degree of decline, a dip, a slip, a tumble or a plunge.

Because the variables in the realms of financial and sporting news are largely standardized and predictable, robots for the most part can be quicker and more accurate than humans, freeing time for journalists to dig deeper and more analytically into stories.  While it is highly unlikely that a robot would have caught the massive Enron accounting fraud, it actually took a long time before one smart human, Bethany McLean, figured it out, too.

So, bring on robo-journalism.  Like chicken soup, it couldn’t hurt.  And it probably will help. 

The newspaper crisis, by the numbers

Roughly a decade after the commercial debut of the Internet, America’s newspapers posted record high advertising sales of $49.4 billion in 2005, leading many publishers to think their businesses would not be seriously affected by the digital revolution. But they were wrong.  

Since hitting that high note in 2005, the industry has undergone a dramatic and traumatic contraction, losing nearly half of its print readership and more than a third of its revenues. With the pre-tax profits of the publicly held publishers cut by 39% since 2003, newsroom staffing has dropped to a historically low level. In spite of the declared determination of most publishers to pivot from print to pixels, the industry's share of the digital advertising market has plunged by more than 50%.   

As illustrated in the following roundup of key performance indicators, it is clear that publishers have failed for the better part of a decade to adequately respond to the new ways that consumers get information and that advertisers want to reach them. 

Because it is easy to become inured to the drip, drip, drip of bad news about newspapers, this quick compendium is offered as a reminder that one of our most valuable journalistic institutions is engaged in a grave, ongoing and so far unresolved crisis. Brace yourself. 

Combined print and digital advertising revenues at the 1,300-plus newspapers in the nation tumbled 55% from $46.2 billion in 2003 to $20.7 billion in 2013, according to the Newspaper Association of America (NAA). Classified print advertising was hit the worst in the 10-year period, plunging by 74% to $4.1 billion in 2013. National print advertising dropped 61% in the decade to not quite $3.1 billion in 2013 and retail print advertising fell 53% to $10.1 billion in 2013. 

Notwithstanding the professed embrace of digital publishing at most newspapers, the industry’s share of the interactive advertising market dropped by 52% in the last decade. While the industry’s collective digital ad revenues rose 181% from $1.2 billion in 2003 to $3.4 billion in 2013, the total digital ad sales in the United States soared 494% from $7.3 billion in 2003 to $42.8 billion in 2013. Accordingly, the newspaper share of the digital advertising market dropped from 16.4% in 2003 to 7.9% in 2013. Newspaper sales figures were provided by the NAA and over-all digital data are from the Interactive Advertising Bureau. 

While newspaper publishers are continuing to gain audience at their web and mobile sites, their interactive efforts typically trail the level of engagement achieved by many native digital media. Here is an example: The NAA reported that a collective 161 million unique individuals visited the digital media of all the newspapers in the land in March, a number equal to two-thirds of American adults. By contrast, Facebook alone attracts 166.5 million uniques per month. Here is the big difference: While the typical visitor spends 1.1 minutes at a newspaper site, the average dwell time at Facebook, the super-sticky social network, is nearly half an hour. 

Weekday print circulation dropped 47% from an average of 54.6 million papers a day in 2004 to an average of 29.1 million papers per day in 2014, according to my analysis of a random sample of data from the Alliance for Audited Media. Sunday circulation in the same period fared somewhat better, sliding 40% to an average of 34.7 million papers per week in the period ended in March, 2014. The circulation decline means that only a quarter of the nation’s 115.2 million households today consume a weekday paper and only 30% of households take Sunday papers. Ten years ago, household penetration was more than 1.6 times higher than it is today. 

NAA data show that the industry’s total advertising and audience revenues across all categories shrank 35% in the last decade, wilting from $57.4 billion in 2003 to $37.6 billion in 2013. The drop occurred even though many publishers sought to offset declines in print advertising and circulation volume by boosting prices for their print products and/or putting paywalls on their digital media. Despite the heavy emphasis most publishers have put on increasing audience revenues, even this category slipped by 3% from $11.2 billion in 2003 to $10.9 billion in 2013.

In spite of effusive efforts by the industry to reduce expenses in the face of plummeting revenues, pre-tax profits of the publicly held newspaper companies fell by 37% in the last 10 years. Earnings before interest, taxes, depreciation and amortization slid from an average of 25.8% in 2003 to 16.3% in the last 12 months, which still compares favorably to the pre-tax margins of 4.9% at Amazon and 9.4% at Walmart. The International News Marketing Association provided the historical data and I compiled the current data at Yahoo Finance. 

One major consequence of the industry-wide contraction is that newsroom staffing dived by 31% from 54,700 journalists in 2002 to 38,000 in 2012, the lowest number since the American Society of News Editors conducted its first newsroom census in 1978. Given additional newsroom reductions at many publications since the ASNE survey, it is fair to assume that staffing is even lower today. In a measure of how coverage has diminished, the number of newspaper reporters covering the nation’s statehouses fell 35% to 470 individuals between 2003 and 2014, according to a survey released last week by the Pew Research Center. 

A newspaper executive told me a few days ago that some people in the industry hate my continuing coverage of the challenges facing newspapers. For the record, I don’t enjoy writing this stuff any more than newspaper people like reading it. But I do it because I am trying to remind them of the urgent and formidable challenges they face in not just protecting their individual businesses but also in preserving the irreplaceable public trust that newspapers represent.  

Newspapers can’t merely dabble at digital

The New York Times wrote the story in 1853 about how Solomon Northup was kidnapped and sold into slavery, but Gawker got most of the page views by publicizing the archived article when “12 Years a Slave” won the Oscar for best picture in 2014. 

This example of how the Times fails to capitalize on its rich content to build digital readership, relevance and revenues came to light in the leak this spring of a candid, unsettling and must-read assessment of the newspaper’s less than elegant effort to pivot from print to pixels. 

Given the size of its staff, the sweep of its ambition and its stature as house organ to the world’s political, economic, academic and cultural elites, the Times is something of a unique case among newspapers. But its struggle to achieve scale and economic sustainability as a full-on digital publisher – as opposed to a legacy newspaper producing digitized renditions of an increasingly superannuated print product – is directly relevant to nearly every newsroom on the globe. 

If you work at a newspaper – or simply care about the health of these important institutions – then you need to read the 90-plus page internal report about how the Times is trying, with less than dazzling success, to retool its culture and business model. The report, which illustrates why digital must be an obsession and not a hobby, is here. 

The document, which never was meant for public disclosure, first was published in May by BuzzFeed – another of the digital interlopers who knows how to generate more page views with a Times story than the Times – on the day after Jill Abramson was ousted as editor of the newspaper. Commissioned by Abramson some months before her exit, the report was written by a crew of journalists headed by no less than A.G. Sulzberger, the son and presumptive successor of publisher Arthur Sulzberger, the head of the family that controls the paper. 

The report details an alarming number of occasions that the Times, like most other newspapers, was out-thought, out-promoted and otherwise out-gunned by the growing phalanx of digital publishers. 

In one painful example, a Huffington Post editor told the Times team that their newspaper was “crushed” by the amount of traffic captured by his site when it repurposed NYT coverage of the death of Nelson Mandela. “I was queasy watching the numbers,” said the unidentified editor quoted in the report. “I’m not proud of this. But this is your competition. You should defend the digital pickpockets from stealing your stuff with better headlines, better social.” 

In another example of digital tone-deafness cited in the report, the author of the sprawling Dasani series on a homeless family trapped in horrific public housing did not get around to tweeting about her own story until two days after the first installment ran. Curiously, noted the report, the newsroom controls the Twitter account but the “business side” runs the Facebook page. 

The barrier to nimble and effective digital publishing at the Times is, as is the case at most other papers, its entrenched print tradition. 

The workflow in the newsroom throughout the day is focused heavily on the print edition, including rewriting the summaries – not the articles themselves – of stories being pitched for the next morning’s front page. “The vast majority of our content is still published late in the evening, but our digital traffic is busiest in the morning,” said the report. “We aim ambitious stories for Sunday because it is our largest print readership, but weekends are the slowest online.” 

Because people who distinguished themselves as writers and editors for the print product hold the senior jobs in the newsroom, they lack the skills, the sensibilities and, frankly, the sass that make for successful digital publishing in an age when readers want their news to be as amusing as it is informative. Because masters of the print universe dominate the top newsroom jobs, many young, digital-savvy staffers depart in despair of ever advancing in the organization. 

While the internal Times report recommends many ways to tinker with priorities, process and personnel, the real problem it identifies – without offering any solution – is the lack of commitment to changing the culture of the institution. In part, the inertia comes from the respect and affection that most of us share for the honorable tradition of print. But it also comes from not understanding that the Times, like all newspapers, has to be willing to aggressively disrupt and reinvent itself before readers and advertisers move on without it.  

Everyone at the Times – and at most other newspapers – has the smarts to do this. But the first step to change is acknowledging that you have a problem. Until there is an inalterable conviction at the Times and other newspapers that they need to overhaul their cultures at Internet speed, they will continue dabbling at digital while fierce and well-financed digital competitors peck them silly. Dabbling won’t be enough.

© 2014 Editor & Publisher