Teradata CTO: 'Black box' deep learning algorithms could limit enterprise use

Matthew Finnegan

teradata conference logo
Image: Teradata

A wide range of businesses are already showing how machine learning can be applied to their applications.

But while there are clear advantages in a variety of fields, automating processes and improving the effectiveness of existing systems, a lack of transparency could make the technology unsuitable for customer-facing systems.

According to Teradata CTO Stephen Brobst, the complexity of deep learning means that it can be extremely difficult to explain why decisions are made by an algorithm.

"One of the disadvantages of deep learning is that it is not very transparent," Brobst said, speaking at Teradata Universe in Nice today. "These multi-layer neural networks with their intermediate data representations are not understandable by the average human, or even the expert human, to be frank."

Brobst referred to a recent IDC report which predicted that, by the end of the decade, a lack of transparency and perceived bias in deep learning algorithms will lead to "backlash" against AI-based services. He said: "This could be a big problem particularly if you are using machine leaning for something that has a consumer-facing impact."

Probst gave the example of deep learning within the healthcare sector, where AI systems could be used to suggest a pathway of care for a patient. This is an ideal application, he said, because deep learning can draw on a wide variety of data to reach conclusions - such as genomic structure, x-ray images and family history.

"But then if [a system] makes the recommendation 'please treat the patient in this way', but you can't explain to the doctor how you got there, or why you should do that, doctors are going to feel very nervous about treating a patient kind of blindly," he said.

While doctors can tie back recommendations to specific research papers, for example, the workings of a neural network are not something that can be easily explained. "For those decisions we need complete transparency," Brobst said. "Deep learning in the current state we are is actually not a good choice."

Another example is within the financial sector. Brobst explained: "Let's say that you are a financial services organisation and you are granting credit or not to an individual based on your deep learning algorithms, when a regulator comes in to say 'well why did you give credit to this person and not that person', [to reply that] 'I don't know, my black box told me so' is not a very good answer. I need to be able to prove that I am not discriminating with inadvertent redlining."

Despite barriers in certain cases, deep learning will have many uses in large organisations, Brobst said.

1  2  Next Page