小密圈精华第一期《云端的威胁检测》和《开源威胁情报的自动整理》

我们整理“安全数据与机器学习”小密圈最近的两篇精华,试运行一期精华总结。第一篇谈了RSA的议题,比如检测Azure上被黑的虚拟主机,第二篇介绍了一篇从开源情报中自动发现和分析IOC的文章,非常有趣。

RSA乱谈:云端的威胁检测

作者:Bindog

首席技术官Mark Russinovich在RSA2017会议上的演讲《Advances in Cloud-Scale Machine Learning for Cyber-Defense》(链接:https://www.youtube.com/watch?v=skSIIvvZFIk),把他的演讲内容要点整理了一下和大家分享:

Mark把攻击方称为红队(Red Team),防御方称为蓝队(Blue Team),红队的Kill Chain是Recon-Delivery-Foothold-Persist-Move-Elevate-Exfiltrate,蓝队的Kill Chain是Gather-Detect-Alert-Triage-Context-Plan-Execute。Mark认为目前所面临的主要有两大挑战,一是False Positives太多(指的是警告太多,难以甄别),二是高度依赖人工分析研判(Kill Chain从Triage开始基本都得靠人来做)。Mark特别指出,可视化并不能解决所有问题,在Flase Positives太多的问题上,可视化基本没啥用,还是得靠引入更多的领域知识来解决。

Mark主要介绍了两个成功在Azure上应用的案例,一是检测Azure上被黑的虚拟主机(尤其是往外发垃圾邮件的主机),方法是综合了Office 365的垃圾邮件数据和Azure的IPFIX数据,利用Gradinet Boosting算法来训练,这个案例主要是想表明不同来源的数据集结合在一起的效果拔群;另一个案例是检测发给用户的钓鱼邮件中的恶意附件,方法是结合反病毒软件扫描结果和沙盒分析结果,其中在沙盒分析中用到了两个模型:Fingerprint模型和Behavioral模型,这个案例主要是想表明将机器学习模型和规则(比如YARA规则)相结合效果拔群。

最后Mark希望蓝队的工作不要停留在发现和检测上,而应该考虑更多检测之外的事情。这里介绍了他微软同事John Lambert的工作,如何把警告数据转变成高价值的情报,请参阅 https://blogs.technet.microsoft.com/johnla/2015/04/26/defenders-think-in-lists-attackers-think-in-graphs-as-long-as-this-is-true-attackers-win/

John Lambert认为:攻击方总是从图的角度思考问题,而防御方总是从表的角度思考问题,这就导致了攻击者总是走在防御者前面,所以作为防御方我们也应该从图的角度去思考问题。怎么理解John的这个思路呢?他的意思是,攻击方往往会利用前期搜集到的情报信息构建一张图(参考Maltego),关键节点和薄弱环节一目了然,在此基础上可以一步步逼近最终目标。而防御方有的只是孤立的警告信息(所谓的表),如某主机流量异常、某某账户发生异常,这些异常过于笼统,并不能直接产生有价值的情报。所以,我们应该把警告(alerts)细化为事件(incidents),例如:p1进程在主机h2上以user3权限得到执行、主机h2上的p5进程向IP地址ip3传输了50M的文件等等,最终把这些事件所涉及的要素串联起来,同样可以生成一张图。通过这个图能更好的理解攻击方的意图,更进一步的,能够利用机器学习算法对图结构进行研判分类,提前截断攻击方的Kill Chain

个人感觉,在云上部署用于威胁检测的机器学习模型是大势所趋,大厂在这方面有着得天独厚的优势:海量数据源和大规模集群,听Mark说他们训练分类恶意附件的模型只需要几分钟,1毫秒以内即可完成分类(个人小作坊的破机器哭晕在厕所)……所以在这方面,去找个靠谱的大腿抱抱是个不错的选择。

皮校长评论:云上判断被侵入机器需要业务知识组合较多,多个维度信息有效融合是重点,搞起来是个细致的工程问题,Azure已经先搞了国内公司要跟上。

碳基体评论:将警告还原成事件,然后根据不同事件中事件主体与事件对象进行关联的思路在很大的数据量的情况下效果尤其明显

PPT下载地址 exp-t11-advances-in-cloud-scale-machine-learning-for-cyber-defense

开源威胁情报的自动整理

作者:tree

从开源情报中自动发现和分析IOC(Indicator of Compromise) 从技术博客中抽取IOC相关的文章,然后利用NLP技术结合安全报告类文章的特点从中自动提取IOC Term和Context并自动生成OpenIOC格式的记录,在大幅降低人工介入的情况下取得了较好的抽取效果。 对2003/1 – 2016/04期间45个安全类博客发布的71000篇文章进行了IOC自动生成和关联分析,从中发现了很多表面看起来没有关联但是实际是有关联的事件。

皮校长评论: 这个方法不错,大家可以看一下增长知识,同时也可以给实战系统做参考。自动分析人工blog的关联给威胁情报补充重要一步,并且还能相互纠错还可以发现长程关联,有自动知识发现的意思。这让我想起来多年前面试彭博社的时候讲过类似的项目,自动综合多家新闻源推测某些金融走势,想法异曲同工。

下载地址:Acing the IOC Game

本文版权由原作者和Kakapo共同拥有。转载须联系kakapo项目组并征得同意,kakapo项目组对任何形式的侵权行为保留追究的权利。详情请咨询help@paperpo.ml。

“小密圈精华第一期《云端的威胁检测》和《开源威胁情报的自动整理》”的34个回复

  1. I wanted to put you one little word to be able to thank you very much once again considering the striking advice you’ve featured at this time. It was certainly seriously open-handed with you to grant freely what most of us would have distributed as an ebook in order to make some dough for their own end, principally considering the fact that you might have done it if you considered necessary. The things additionally served as a great way to fully grasp that most people have the same dream really like my own to find out way more in terms of this condition. I am sure there are a lot more pleasant moments up front for those who read carefully your website.

  2. I’m writing to make you be aware of what a fantastic discovery my wife’s daughter found reading through the blog. She learned plenty of pieces, not to mention what it’s like to possess an ideal coaching mindset to have certain people just know certain very confusing topics. You actually surpassed visitors’ desires. Thanks for showing such precious, trusted, educational and as well as unique tips on this topic to Gloria.

  3. I want to show my thanks to you for bailing me out of such a predicament. Right after surfing through the internet and getting solutions which were not pleasant, I thought my entire life was done. Living without the strategies to the issues you have sorted out by means of your main write-up is a critical case, as well as the kind that could have in a wrong way damaged my entire career if I had not come across the website. The understanding and kindness in playing with every aspect was crucial. I am not sure what I would have done if I had not discovered such a step like this. I’m able to at this moment relish my future. Thanks for your time so much for this reliable and results-oriented help. I won’t hesitate to endorse your web page to any person who would need direction on this issue.

  4. I really wanted to jot down a comment in order to thank you for those superb steps you are posting at this site. My long internet investigation has finally been compensated with awesome concept to share with my relatives. I would state that that many of us site visitors actually are extremely endowed to live in a good website with very many perfect individuals with great guidelines. I feel quite blessed to have seen your web page and look forward to tons of more brilliant minutes reading here. Thank you once more for a lot of things.

  5. I want to show appreciation to you just for rescuing me from this situation. After browsing through the world-wide-web and obtaining opinions that were not beneficial, I assumed my life was well over. Existing devoid of the answers to the problems you have fixed by way of your posting is a serious case, and ones that would have in a negative way affected my entire career if I had not come across your site. Your personal know-how and kindness in touching all the pieces was important. I don’t know what I would have done if I hadn’t discovered such a point like this. It’s possible to now relish my future. Thanks a lot very much for your specialized and effective help. I won’t hesitate to suggest the sites to anybody who wants and needs recommendations about this area.

  6. Thanks for each of your efforts on this web page. Kim really loves going through research and it’s really easy to see why. All of us notice all relating to the powerful manner you render very helpful tactics via this blog and as well as encourage response from the others about this issue while our favorite princess is without question discovering so much. Have fun with the rest of the year. You’re doing a powerful job.

  7. I in addition to my friends have been looking at the good things on your web blog then immediately came up with an awful feeling I had not thanked the web site owner for those secrets. These boys were definitely totally excited to see them and have now undoubtedly been having fun with these things. Thanks for simply being well thoughtful and then for finding this sort of remarkable things millions of individuals are really eager to learn about. My personal honest regret for not expressing appreciation to earlier.

  8. I and my pals were found to be going through the excellent secrets from your web site and so all of a sudden developed a horrible suspicion I never thanked the site owner for those tips. These guys became certainly excited to study them and now have quite simply been loving those things. Thank you for genuinely really kind and also for deciding on this kind of beneficial resources millions of individuals are really needing to know about. My sincere regret for not saying thanks to sooner.

  9. This site is mostly a stroll-by way of for the entire information you wished about this and didn抰 know who to ask. Glimpse right here, and also you抣l definitely discover it.

  10. I simply wanted to appreciate you again. I’m not certain the things I would have achieved without these secrets discussed by you concerning that field. It was before a real frustrating concern for me, nevertheless observing the very professional manner you solved that took me to cry over joy. Now i am grateful for this assistance and then expect you know what a powerful job that you are getting into instructing some other people thru your blog. Probably you haven’t encountered all of us.

  11. I抦 impressed, I must say. Really rarely do I encounter a weblog that抯 both educative and entertaining, and let me let you know, you will have hit the nail on the head. Your concept is excellent; the difficulty is something that not enough individuals are talking intelligently about. I’m very happy that I stumbled across this in my search for something referring to this.

  12. I as well as my pals were actually studying the excellent guidelines found on your web blog then immediately I got a terrible feeling I had not expressed respect to the web site owner for them. These boys came consequently happy to read all of them and have now seriously been tapping into these things. Thanks for indeed being indeed considerate and for obtaining such tremendous areas millions of individuals are really desperate to be informed on. Our sincere apologies for not expressing gratitude to sooner.

  13. There are some fascinating closing dates on this article however I don抰 know if I see all of them heart to heart. There’s some validity however I will take hold opinion till I look into it further. Good article , thanks and we would like extra! Added to FeedBurner as effectively

  14. Thanks so much for providing individuals with an extremely memorable opportunity to check tips from here. It is always so sweet and packed with a lot of fun for me and my office mates to search your website at the very least thrice a week to study the latest tips you have got. And definitely, I’m certainly impressed considering the amazing knowledge you give. Selected 2 areas in this post are unquestionably the most impressive we have all had.

  15. I would like to show some thanks to this writer for bailing me out of this issue. Just after surfing around throughout the world wide web and obtaining tricks which are not helpful, I figured my life was gone. Living without the answers to the issues you have solved by means of your short post is a crucial case, and the ones which could have in a negative way affected my entire career if I hadn’t come across your site. The mastery and kindness in maneuvering all the pieces was valuable. I don’t know what I would’ve done if I had not come upon such a thing like this. I am able to at this moment relish my future. Thanks for your time so much for your impressive and amazing guide. I won’t hesitate to recommend your web blog to anyone who desires care on this area.

  16. Can I just say what a reduction to seek out someone who truly knows what theyre speaking about on the internet. You undoubtedly know find out how to convey a problem to light and make it important. Extra folks need to learn this and perceive this side of the story. I cant consider youre not more in style since you positively have the gift.

  17. I want to express thanks to this writer just for rescuing me from this situation. Just after researching throughout the world-wide-web and obtaining solutions which were not productive, I was thinking my life was over. Existing minus the strategies to the issues you’ve fixed by means of your write-up is a crucial case, and the kind that might have badly damaged my entire career if I had not come across your blog. Your personal understanding and kindness in playing with all things was important. I’m not sure what I would’ve done if I had not discovered such a stuff like this. I am able to at this time relish my future. Thanks so much for this expert and sensible help. I won’t be reluctant to refer your blog post to any individual who should have counselling on this situation.

  18. The subsequent time I learn a weblog, I hope that it doesnt disappoint me as a lot as this one. I imply, I know it was my choice to learn, but I really thought youd have one thing fascinating to say. All I hear is a bunch of whining about one thing that you can fix if you happen to werent too busy looking for attention.

  19. I抦 impressed, I need to say. Really not often do I encounter a weblog that抯 both educative and entertaining, and let me tell you, you could have hit the nail on the head. Your idea is outstanding; the difficulty is something that not sufficient people are talking intelligently about. I am very pleased that I stumbled throughout this in my search for one thing regarding this.

  20. Thanks so much for giving everyone an exceptionally nice opportunity to read in detail from here. It can be so awesome and stuffed with amusement for me personally and my office colleagues to search the blog minimum three times per week to study the fresh items you have got. And indeed, I am also actually impressed with your surprising techniques you give. Certain 1 tips in this posting are basically the best I’ve ever had.

  21. I was more than happy to find this net-site.I wanted to thanks in your time for this glorious learn!! I positively having fun with each little little bit of it and I have you bookmarked to check out new stuff you weblog post.

  22. Oh my goodness! an amazing article dude. Thanks Nonetheless I am experiencing difficulty with ur rss . Don抰 know why Unable to subscribe to it. Is there anybody getting similar rss downside? Anyone who knows kindly respond. Thnkx

  23. Needed to write you this little word to help give thanks yet again for those pretty tricks you’ve featured on this website. This has been simply seriously generous with you giving extensively all many individuals might have advertised for an electronic book to make some profit on their own, specifically since you could have done it in case you desired. The advice likewise acted to become a easy way to comprehend other people online have the same passion like mine to figure out lots more around this issue. I am sure there are lots of more pleasant periods in the future for individuals who scan your blog.

  24. There are some interesting time limits on this article however I don抰 know if I see all of them heart to heart. There is some validity however I’ll take maintain opinion until I look into it further. Good article , thanks and we want more! Added to FeedBurner as well

发表评论

电子邮件地址不会被公开。 必填项已用*标注