AI and ML in Cybersecurity Part 1: Don’t Believe the Hype! …Yet

Dumb AI

If you listen at all to industry news, you can’t escape the onslaught of marketing hype surrounding Artificial Intelligence (AI) and Machine Learning (ML).  Self-driving cars!  Intelligent IT security!  Is SkyNet ready to take over the world?  Not quite.  Despite the optimism and promise, AI and ML largely remain in an immature stage where they suffer mistakes and biases from classic programming issues, which we will explore below:  garbage-in, garbage-out and poor quality control.  Still, with the potential for huge benefits on the horizon, we cannot simply reject security tools that have tried to incorporate these advances.  Instead, we need to create a strategy for adoption that provides a safety net for failure through overlapping technology – just as we do for any other security product.

AI will be eventually be good.  Brilliant minds are working on it.  In fact, we at Soliton are working on machine learning algorithms ourselves. To get the inside story on how to overcome weaknesses in ‘smart’ security solutions, we asked our own head of AI, Dr. Tedd Hadley: Why do AI and ML seem to have problems?

Tedd: In traditional software, the designer is given input and output requirements and he or she writes a program that will do the transformation.  But in AI/ML, the designer is given input and output requirements and learning algorithms chosen by the designer write the program that will do the transformation.  Put another way, the designer in the traditional case learns the transformation needed between input and output and laboriously translates that knowledge into C++, Python, Java, etc.  In AI/ML, the designer passes inputs and outputs to a learning algorithm and the algorithm automatically figures out the transformations needed to get the desired output using back-propagation/gradient-descent, which tediously sets and fine-tunes potentially millions of parameters.   These parameters, for an artificially intelligent learning algorithm, are analogous to a traditional software program.

OK, so how does that explain the classic examples for garbage-in, garbage-out problems that reveal themselves as biases or vulnerability to manipulation:

  • In 2016, Twitter users famously taught Microsoft’s Tay.ai chatbot to be “a racist ***hole in less than a day” as described by James Vincent of the Verge.
  • In 2017, The Guardian profiled a trio of AI biases:
    • Hannah Devlin revealed how algorithms that could interpret everyday language reflected the same gender and racial biases present in humans.
    • Stephen Buranyi covered how programs attempting to eliminate human biases in predictions strongly mirrored human biases.
    • Naaman Zhou profiled how Volvo’s self-driving cars were confused by Kangaroos – not that confusion is limited to AI since the marsupials are responsible for 90% of the animal-vehicle collisions in Australia.
  • In 2018, Andy Norton, SC Media Executive Insight Blogger, summarized the findings from SignedMalware.org that revealed a bias in malware detection. Researchers manipulated AI’s verdicts on malware by providing certificates or polluting the existing ratings on VirusTotal!

Tedd: Let’s talk software bugs.  It is quite conceivable that a software program written in the traditional way by a human might be discovered to cheat by relying on code-signing to classify malware, or accidentally leave out kangaroos in large-mammal detection vehicle software, or even use racist language in chat sessions. We would, however, rightly question the competence of the company’s Quality Assurance (QA) department, their software development experience and methodology, and their developer hiring practices.  Modern software development recognizes that programmers make mistakes, miscommunications occur, vague specifications happen, and that there are bad apples.  But over time solid strategies have been developed to minimize that.  In the AI bias cases, good traditional QA for malware should discover the code-signing trick and send the software back for fixing; traditional QA for large-mammal detection software likely makes use of at least one mammal specialist who is aware of the problem of kangaroos near Australian roads and discovers the flaw…

So why are we somehow bypassing the quality controls necessary to avoid these situations?  Come back next week as we’ll go into detail on how to overcome the inherent weaknesses that occur as a result of biases.  In the meantime, we urge everyone to maintain healthy skepticism regarding the capabilities of new technology. As you seek to secure your systems, overlap this new technology with a foundation of basic technology that has proven capabilities, so that any unforeseen critical errors will not result in catastrophic damage.

 

https://www.theverge.com/2016/3/24/11297050/tay-microsoft-chatbot-racist

https://www.theguardian.com/technology/2017/apr/13/ai-programs-exhibit-racist-and-sexist-biases-research-reveals

https://www.theguardian.com/inequality/2017/aug/08/rise-of-the-racist-robots-how-ai-is-learning-all-our-worst-impulses

https://www.theguardian.com/technology/2017/jul/01/volvo-admits-its-self-driving-cars-are-confused-by-kangaroos

https://www.scmagazine.com/artificial-intelligence-has-a-dirty-little-secret–signature-addiction/article/742246/