Fair and legitimate point, but is there a practical and better way? It's not akin to a company testing its own applications (although if we take mobile apps as a hint, we're not exactly getting an A+ there, either.
Microsoft has been legendary in its crowd-sourcing strategy: An initial software cut is released to millions, and they find the holes. This gave rise to my favorite Microsoft quip, many years old and unattributable at this point, unfortunately: "Here at Microsoft, quality is Job 1.1." The crazy thing is that it generally worked. How did Heartbleed spend two years in full circulation before any security researcher noticed this error?
Some are convinced that the hole must have been noticed by someone. The National Security Agency has been accused of knowing of this hole and exploiting it. The accusation led to the NSA issuing what may be the least credible denial in quite some time: "Reports that NSA or any other part of the government were aware of the so-called Heartbleed vulnerability before April 2014 are wrong," the statement from the U.S. Office of the director of National Intelligence said. "The Federal government was not aware of the recently identified vulnerability in OpenSSL until it was made public in a private sector cybersecurity report."
There are two parts of the full statement ( read it here) where credibility leaches out. First, between the CIA, the FBI, the NSA, the military and let's say 200 other government operations, it's ludicrous to declare that nobody knew about something. How do you know that one Army security specialist didn't know? Not every geeky hole that is discovered is necessarily included in a memo to senior management. Had they said "to the best of our knowledge" or "we can't confirm that anyone here knew about it," that would at least be plausible. It's like my teenager telling me that nobody in her high school uses drugs or drinks. It doesn't pass the laugh test because there is no way she could know such information definitively.
The second concern with the NSA statement is the very last line: "Unless there is a clear national security or law enforcement need, this process is biased toward responsibly disclosing such vulnerabilities." Nothing in the statement says that no such need was found in this case. This is akin to my daughter following up her no-drugs testimony by saying, "I will always tell you the absolute truth about such things, unless I conclude that it would cause problems for my friends, in which case I would lie."
I'm generally no fan of adding bureaucracy, but it might be time to create formal review procedures — ideally, with multiple layers — with people actively and openly looking for holes. Peer review is great, but for anything as mission-critical as Internet security, we are way past the time to proactively seek out such holes, rather than hoping we stumble upon them.