On December 20, 1998, an internal CIA memo was sent by a field agent about a missed opportunity to "hit" Osama bin Laden while he was reportedly visiting a mosque near Kandahar, Afghanistan. "I said hit him tonight; we may not get another chance," CIA agent Gary Schoen wrote. "We may well come to regret the decision not to go ahead."

The memo was sent to to Michael Scheuer, then head of the CIA's Osama bin Laden "station," and is one of more than 100 documents declassified and published by the National Security Archive this week. Although some have been previously cited or quoted in the Report of the 9/11 Commission, the raw documents themselves illustrate the frustrations and missteps in the hunt for Osama bin Laden and alarm among some at the CIA about al Qaeda's growing sophistication and its plans for attacking U.S. interests.

Scheuer replies to Schoen the following day. "This is the third time you and your officers have put UBL in this government's sights and they have balked each time at doing the job. ... They spent a good deal of time yesterday worrying that some stray shapnel might hit the Habash mosque and 'offend' Muslims."

The documents have been heavily redacted before publication. But they give a sense of the aggravation, the guesswork and a never-ending sense of crisis in the intelligence community's pursuit of the al Qaeda leader before his eventual discovery and killing in May 2011. They also show that co-operation between different agencies was improving before 9/11 but that gaps and different priorities hampered counter-terrorism efforts.

More: Bin Laden's death

Another memo from Scheuer, in May 1999, complained: "For the past forty months the CIA, and especially the do (Directorate of Operations) has been in this endeavor virtually alone. ... until the african bombings [of the US embassies in Kenya and Tanzania in August 1998] the u.s. military did not believe that ubl was a problem."

"Having a chance to get ubl three times in 36 hours and foregoing the chance each time has made me a bit angry," he writes.

By 1999, the Counterterrorism Center at the CIA had built up an impressive database on al Qaeda, and developed a healthy respect for its growing capabilities and meticulous planning. An internal study produced after the embassy attacks showed that al Qaeda had begun preparations to bomb the Nairobi mission fully five years before the actual attack. And a report in January 1999 — widely circulated within the Clinton administration — described al Qaeda as similar to a "global criminal syndicate" with a complex organizational structure and a presence in some 60 countries.

The search for bin Laden wasn't immune from the cycle of politics and budgets, either. In May 2000, the head of the bin Laden unit warned: "Need forward movement on supplemental soonest due to expected early [Congressional] recess due to conventions, campaigning and elections. Due to budgetary constraints ... CTC/UBL [Counterterrorism Center/Osama bin Laden Unit] will move from offensive to defensive posture."

From the archives: The 9/11 attacks

There are also exchanges in the documents about the search for al Qaeda operatives in 2001, with scant information about their identities and confusion about their names. A secret memo from May 2001 details the fruitless search for a man named Khallad who had been in Bangkok the previous year. "Then he, what, went to Bangkok and disappeared? Anyway after the Cole bombing [the attack in Aden, Yemen, on the USS Cole in October 2000) we had another look at the pics, thinking Khalid might have been Khallad. He wasn't."

The exchange among unnamed officials continued with one observing: "I'm either missing something or someone saw something that wasn't there."

It appears that "Khallad" — spelled a multitude of different ways — was likely Khalid al-Mihdhar, who would become one of the 9/11 hijackers. He had left Bangkok in January 2000 and flown to Los Angeles.

The memos illustrate the growing sense of unease in the CIA about al Qaeda plans to attack American interests. By June 2001 a top secret memo notes that an Arabic TV news channel had reported bin Laden had gone into hiding and said: "Multiple reports indicate that extremists [redacted] expect Bin Ladin (sic) to launch attacks over the coming days, possibly against Israeli or US interests."

But there is little sense — at least in the heavily redacted memos that have been published — that the attack would be inside the United States, with another memo suggesting "the Arabian peninsula as a likely venue for an anti-US attack." An update from July 3, 2001, says that "within the Gulf, Saudia Arabia is the most likely venue." But there is plenty of uncertainty, with another July 2001 memo suggesting that an al Qaeda attack had been delayed a few months.

A year after bin Laden's death

Also among the declassified documents is a secret report by the CIA Inspector General on the CIA's Counterterrorist Center (CTC) — dated just two weeks before the 9/11 attacks. The report concluded that the CTC was a "well-managed component that successfully carries out the Agency's counterterrorist responsibilities to collect and analyze intelligence." But it also spoke of information overload, noting that "some consumers worry that Center analysts do not have the time to spot trends or knit together the threads." And it noted "employee burnout," adding the Center had "difficulty in attracting a sufficient cadre of trained, experienced officers."

Even so, the CTC's "Usama bin Laden Station was frequently singled out for its exceptional support, with its work desccibed as "excellent" and "timely." The station had been established in 1996, when bin Laden was still living in Sudan.

Notably, the Inspector General's report included a warning that the long-term discipline needed in analyzing terror groups "can be difficult to maintain in the current atmosphere, which rewards instant results." Some analysts had complained about a "constant state of crisis" that limited their ability to develop in-depth expertise. One analyst "worried that he only had time to answer the mail, and as a result he might miss warning signs of a threat."

In 2004, a top secret report from the Director of Central Intelligence, now declassified but heavily redacted, offered a ferocious defense of the work of the CIA in warning policy-makers about the threat of al Qaeda in the years before 9/11. But it, too, acknowledged that analysts had been "consumed by tactical work. ... CIA therefore created a Strategic Analytic Unit" to fix the problem. But that was only in July 2001, weeks before the September attacks.

The report also details the use of Predator drones to search for bin Laden, with the first mission on September 7, 2000. But there were concerns that the drones were vulnerable to detection and the U.S. Air Force "notified the CIA it would have to pay for lost aircraft." Twice in the fall of 2000, a Predator "observed an individual most likely to bin Ladin; however we had no way at the time to react to this information."

Though they could not know it, time was running out. In December 2000, the CIA submitted a new plan in its targeting of al Qaeda "that would have significantly expanded our activities. ... It was too late for the departing Clinton administration to take action on this strategic proposal."

The same report speaks of excellent collaboration between the CIA and FBI on numerous terror cases, but then adds: "A major, ongoing concern is FBI's own internal dissemination system. CIA officers still often find it necessary to hand-deliver messages to the intended recipient within the FBI." And it laments "the loss of potential intelligence opportunities because of deference to law enforcement goals." In other words: at what point do you make an arrest or close down a terror conspiracy?

And in a passage with echoes today, there is a complaint about leaks compromising the Agency's ability to implement what became known as "the Plan" to get bin Laden. "Persistent publicity and leaks of information about our methods in the United States and abroad caused the terrorists to [redacted] emphasize their compartmentation."