Home · Blog · Projects

Michael Henriksen Michael Henriksen

http://michenriksen.com/ michenriksen.com 2017-07-21T13:37:00Z Michael Henriksen http://michenriksen.com/ tag:michenriksen.com,2017-07-21:/blog/subdomain-takeover-detection-with-aquatone/ Subdomain takeover detection with AQUATONE 2017-07-21T13:37:00Z 2017-07-21T13:37:00Z <%= render '/post_head.html', :post => item %> <p>Hostile subdomain takeover is a very prevalent and potentially critical security issue. It's a well-known attack vector and easy to exploit, and should therefore be taken seriously.</p> <p>A subdomain takeover vulnerability typically happens when an organization assigns a subdomain to an external service, e.g. a support ticketing system like <a href="https://zendesk.com/">Zendesk</a>, a cloud application platform like <a href="https://www.heroku.com/">Heroku</a> or maybe a content delivery network like <a href="https://www.fastly.com/">Fastly</a>. Maybe the organization is only assessing the service, or maybe they switch to a different service, but for some reason the organization later decides to delete their account on the service, but forget one important step: to remove the subdomain DNS configuration to the service.</p> <p>Having a dangling subdomain pointing to an unused external service leaves it open for takeover and complete control of an attacker, as they simply need to sign up to the same service and claim the dangling subdomain. Now they control the content on the subdomain which they can use to launch phishing attacks, bypass security controls and other mischief.</p> <p><a href="https://detectify.com/">Detectify</a> wrote a <a href="https://labs.detectify.com/2014/10/21/hostile-subdomain-takeover-using-herokugithubdesk-more/">blog post</a> about subdomain takeover back in 2014, if you're interested in knowing more about this attack vector.</p> <h2>Detecting subdomain takeovers with AQUATONE</h2> <p>In case you don't know, I recently released <a href="/blog/aquatone-tool-for-domain-flyovers/">AQUATONE</a> which is a toolset for doing subdomain discovery, port scanning and screenshotting. Check out the <a href="/blog/aquatone-tool-for-domain-flyovers/">blog post</a> for more information.</p> <p>A new addition to the AQUATONE toolset is <code>aquatone-takeover</code> which can detect potential subdomain takeover issues across a bunch of popular external services:</p> <div class="screenshot"> <a class="fancybox" href="/images/aquatone/takeover.png" title="Finding subdomains vulnerable to hostile takeover..."> <img class="thumbnail" src="/images/aquatone/takeover_thumbnail.png" alt="" /></a> <small>Finding subdomains vulnerable to hostile takeover...</small> </div> <p>To demonstrate the functionality of aquatone-takeover, I temporarily configured a couple of subdomains on <code>michenriksen.com</code> pointed at 3 different external services. aquatone-takeover detects all three as potentially vulnerable:</p> <ul> <li><code>assets.michenriksen.com</code> is a <code>CNAME</code> record pointing to an unclaimed <a href="https://aws.amazon.com/s3/">Amazon S3</a> bucket.</li> <li><code>store.michenriksen.com</code> is a <code>CNAME</code> record pointing to <a href="https://www.shopify.com/">Shopify</a>, a popular ecommerce platform, and the subdomain has not been registered with any account.</li> <li>Finally, <code>help.michenriksen.com</code> is a <code>CNAME</code> record pointing at a non-existant account on <a href="https://www.desk.com/">Desk</a>, a popular support ticketing system.</li> </ul> <p>These are not the only external services that aquatone-takeover can detect, it finds subdomain takeover vulnerabilities across - at the time of writing - a total of 25 services:</p> <ul> <li><a href="https://aws.amazon.com/s3/">Amazon S3</a> (Cloud storage)</li> <li><a href="https://www.zendesk.com/">Campaign Monitor</a> (Email marketing)</li> <li><a href="https://cargocollective.com/">Cargo</a> (Web publishing platform)</li> <li><a href="https://aws.amazon.com/cloudfront/">Cloudfront</a> (Content delivery network)</li> <li><a href="https://www.desk.com/">Desk</a> (Customer service and helpdesk ticket software)</li> <li><a href="https://www.fastly.com/">Fastly</a> (Content delivery network)</li> <li><a href="https://feed.press/">FeedPress</a> (Feed analytics and Podcast hosting)</li> <li><a href="https://freshdesk.com/">Freshdesk</a> (Customer support software and ticketing system)</li> <li><a href="https://ghost.org/">Ghost</a> (Publishing platform)</li> <li><a href="https://pages.github.com/">GitHub Pages</a> (GitHub static website hosting)</li> <li><a href="https://www.helpscout.net/">Help Scout</a> (Customer service software and education platform)</li> <li><a href="https://helpjuice.com/">Helpjuice</a> (Knowledge base software)</li> <li><a href="https://www.heroku.com/">Heroku</a> (Cloud application platform)</li> <li><a href="https://instapage.com/">Instapage</a> (Landing page platform)</li> <li><a href="https://www.pingdom.com/">Pingdom</a> (Website and performance monitoring)</li> <li><a href="https://www.shopify.com/">Shopify</a> (Ecommerce platform)</li> <li><a href="https://www.statuspage.io/">StatusPage</a> (Status page hosting)</li> <li><a href="https://www.surveygizmo.com/">SurveyGizmo</a> (Online survey software)</li> <li><a href="https://www.teamwork.com/">Teamwork</a> (Project management, help desk and chat software)</li> <li><a href="https://tictail.com/">Tictail</a> (Social shopping platform)</li> <li><a href="https://www.tumblr.com/">Tumblr</a> (Microblogging and social networking platform)</li> <li><a href="https://unbounce.com/">Unbounce</a> (Landing page builder and conversion marketing platform)</li> <li><a href="https://www.uservoice.com/">UserVoice</a> (Product management software)</li> <li><a href="https://wpengine.com/">WPEngine</a> (WordPress blog hosting)</li> <li><a href="https://www.zendesk.com/">Zendesk</a> (Customer service software and support ticket system)</li> </ul> <p>I hope to expand this list with many more services, so please let me know if you have any ideas or go ahead and contribute more detector modules on <a href="https://github.com/michenriksen/aquatone">GitHub</a>.</p> <p>Give aquatone-takeover a try! You can install the AQUATONE toolset with <code>gem install aquatone</code> or get the latest version with <code>gem update aquatone</code> if you already have it installed.</p> <p>Here's a small list of articles and reports on subdomain takeover issues across the web:</p> <ul> <li><a href="https://www.grahamcluley.com/hacker-defaces-donald-trump-fundraising-site-via-subdomain-takeover-attack/">Hacker defaces Donald Trump fundraising site via subdomain takeover attack</a></li> <li><a href="https://hackernoon.com/subdomain-takeover-of-blog-snapchat-com-60860de02fe7">Subdomain takeover of blog.snapchat.com</a></li> <li><a href="https://hackerone.com/reports/207576">Subdomain takeover on s3.shopify.com</a></li> <li><a href="https://hackerone.com/reports/186766">Subdomain takeover on happymondays.starbucks.com due to non-used AWS S3 DNS record</a></li> <li><a href="https://www.arneswinnen.net/2017/06/authentication-bypass-on-ubers-sso-via-subdomain-takeover/">Authentication bypass on Uber’s Single Sign-On via subdomain takeover</a></li> </ul> <p><strong>Happy hunting!</strong></p> tag:michenriksen.com,2017-06-18:/blog/geolocating-miriam-steimer/ Geolocating Miriam Steimer 2017-06-18T13:37:00Z 2017-06-18T13:37:00Z <%= render '/post_head.html', :post => item %> <p>If you have been following my blog, you might know that one of my weird hobbies is to determine the location of where pictures were taken. My previous post on <a href="/blog/geolocating-maks-czuperski/">Geolocating Maks Czuperski</a> caught quite a lot of buzz, so I thought I would write up how I solved another recent geolocation challenge, as it seems like you don't necessarily need to catch the <a href="https://www.bellingcat.com/news/uk-and-europe/2015/10/08/mh17-the-open-source-evidence/">Russian Ministry of Defense lying</a> about passenger planes being shot out of the sky for people to be interested in this sort of stuff.</p> <p>It all started when I saw a Retweet from <a href="https://twitter.com/trbrtc/status/869518189738299394">Christiaan Triebert</a> on my Twitter feed:</p> <blockquote class="twitter-tweet tw-align-center" data-lang="en"><p lang="en" dir="ltr"><a href="https://twitter.com/trbrtc">@trbrtc</a> where are we? <a href="https://twitter.com/hashtag/geolocation?src=hash">#geolocation</a> <a href="https://twitter.com/hashtag/itsagame?src=hash">#itsagame</a> <a href="https://t.co/g8JDCgw9Af">pic.twitter.com/g8JDCgw9Af</a></p>&mdash; miriam steimer (@miriamsteimer) <a href="https://twitter.com/miriamsteimer/status/869506933933387777">May 30, 2017</a></blockquote> <script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script> <p>The tweet from <a href="https://twitter.com/miriamsteimer">Miriam Steimer</a> seemed to challenge Christiaan Triebert to find out where they were to come join them for lunch. It didn't exactly look like a challenge meant for everyone but I couldn't ignore it, and so I decided to give it a go...</p> <p>Right-clicking on the image and <em>Copy Image Address</em> gave me a pretty high resolution version of the image where I could pick out details easier.</p> <div class="screenshot"> <a class="fancybox" href="/images/geolocation2/wherearewe.jpg" title="The picture from Miriam Steimer's tweet"> <img class="thumbnail" src="/images/geolocation2/wherearewe_thumbnail.jpg" alt="" /></a> <small>The picture from Miriam Steimer's tweet (<a href="/images/geolocation2/wherearewe_large.jpg" target="_blank">click here</a> for high resolution)</small> </div> <p>My first idea was to comb the picture for any <em>easy-wins</em> such as the name of the place on the yellow parasols in the background, the napkins on the table or perhaps on the cutlery or plates, but no luck there. It did make me a bit hungry though...</p> <p>One detail that I noticed was something that looked like a typical employee badge hidden under the phone on the middle table:</p> <div class="screenshot"> <a class="fancybox" href="/images/geolocation2/wherearewe_zoom_badge.jpg" title="A possible employee badge under the phone."> <img class="thumbnail" src="/images/geolocation2/wherearewe_zoom_badge_thumbnail.jpg" alt="" /></a> <small>A possible employee badge under the phone.</small> </div> <p>The badge made me think that maybe they were having lunch at a work cafeteria or similar, so time to figure out where Miriam works...</p> <p>Figuring out where she works was pretty straight-forward as her Twitter profile bio says she is a journalist working for <a href="https://www.zdf.de/">ZDF</a> and that she is located in Mainz, Germany.</p> <div class="screenshot"> <a class="fancybox" href="/images/geolocation2/miriam_steimer_profile.png" title="Miriam's Twitter profile."> <img class="thumbnail" src="/images/geolocation2/miriam_steimer_profile.png" alt="" /></a> <small>Miriam's Twitter profile.</small> </div> <p>With a bit of Googling, I learned that ZDF has a big broadcasting center in Mainz with pretty interesting architecture:</p> <div class="screenshot"> <a class="fancybox" href="/images/geolocation2/maps_zdf_sendezentrum.png" title="Google Maps view of the ZDF Sendezentrum in Mainz."> <img class="thumbnail" src="/images/geolocation2/maps_zdf_sendezentrum_thumbnail.png" alt="" /></a> <small>Google Maps view of the <em>ZDF Sendezentrum</em> in Mainz.</small> </div> <p>I went back to the picture from Miriam's tweet to see if I could find anything that I could use to confirm that the picture was taken somewhere at the ZDF broadcasting center. After a bit of staring, I noticed that a building facade with something that looks like shiny metallic plates is visible in the background. This could be useful for determining the location more precisely.</p> <div class="screenshot"> <a class="fancybox" href="/images/geolocation2/wherearewe_background.jpg" title="A building facade in the background."> <img class="thumbnail" src="/images/geolocation2/wherearewe_background_thumbnail.jpg" alt="" /></a> <small>A building facade in the background.</small> </div> <p>Scrolling through the Google Image search results for <em>ZDF sendezentrum</em> made me come across a <a href="http://www.jobundfit.de/uploads/pics/Cafeteria_SZ_1_Ausgabelinie.jpg">picture</a> of something that looked very much like a cafeteria:</p> <div class="screenshot"> <a class="fancybox" href="/images/geolocation2/casino_cafeteria.jpg" title="Casino Cafeteria."> <img class="thumbnail" src="/images/geolocation2/casino_cafeteria.jpg" alt="" /></a> <small>Casino Cafeteria.</small> </div> <p>The picture was found on a <a href="http://www.jobundfit.de/qualitaetsstandard/zertifizierung/praxispaten/zdf-casino-mainz.html">web page</a> about <em>ZDF-Casino GmbH</em> catering company getting the <em>JOB &amp; Fit</em> certification. Doing another Google Image search for <em>ZDF sendezentrum casino</em> I came across another valuable <a href="http://www.stepmap.de/landkarte/sendezentrum-zdf-1267042.png">image</a>:</p> <div class="screenshot"> <a class="fancybox" href="/images/geolocation2/sendezentrum_map.png" title="A map of the ZDF Sendezentrum with the Casino building."> <img class="thumbnail" src="/images/geolocation2/sendezentrum_map_thumbnail.png" alt="" /></a> <small>A map of the ZDF Sendezentrum with the Casino building.</small> </div> <p>Knowing that the name of the ZDF cafeteria is called Casino and having a map of the area was good and all, but I still needed to find a building with the facade that is visible in the picture. I scrolled through some more Google Image search results and came across another <a href="http://realschule-plus-langenlonsheim.com/s/cc_images/cache_2477728153.JPG">picture</a> that caught my attention:</p> <div class="screenshot"> <a class="fancybox" href="/images/geolocation2/sendezentrum_entrance.jpg" title="Entrance to the ZDF Sendezentrum with directions and an interesting building in the background."> <img class="thumbnail" src="/images/geolocation2/sendezentrum_entrance_thumbnail.png" alt="" /></a> <small>Entrance to the ZDF Sendezentrum with directions and an interesting building in the background.</small> </div> <p>The picture showed what looks like the entrance to the ZDF Sendezentrum with directions to the <em>Kasinogebäude</em> and more importantly, a building in the background with what looks like the same kind of shiny metallic plates.</p> <p>I wrote a quick tweet to Miriam and Christiaan to figure out if I was going in the right direction:</p> <blockquote class="twitter-tweet tw-align-center" data-conversation="none" data-lang="en"><p lang="en" dir="ltr">Hey, random twitterer here, looks like it could be somewhere near the &quot;Kasinogebäude&quot;?</p>&mdash; Michael Henriksen (@michenriksen) <a href="https://twitter.com/michenriksen/status/869559591419478016">May 30, 2017</a></blockquote> <script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script> <p>After a while, Christiaan replied:</p> <blockquote class="twitter-tweet tw-align-center" data-conversation="none" data-lang="en"><p lang="en" dir="ltr">Correct! Should not be hard to find the exact location now — good luck ;-)</p>&mdash; Christiaan Triebert (@trbrtc) <a href="https://twitter.com/trbrtc/status/869563326073888768">May 30, 2017</a></blockquote> <script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script> <p>Perfect! I used the map picture I found earlier to align my Google Maps view and tweeted my guess:</p> <blockquote class="twitter-tweet tw-align-center" data-conversation="none" data-lang="en"><p lang="en" dir="ltr">Somewhere in the red square, perhaps? <a href="https://t.co/c3HH84PPbs">pic.twitter.com/c3HH84PPbs</a></p>&mdash; Michael Henriksen (@michenriksen) <a href="https://twitter.com/michenriksen/status/869567449733234690">May 30, 2017</a></blockquote> <script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script> <p>Christiaan replied again with their exact location:</p> <blockquote class="twitter-tweet tw-align-center" data-conversation="none" data-lang="en"><p lang="en" dir="ltr">Nice, very close, just a few more meters northward! <a href="https://t.co/qo9zk86IAH">pic.twitter.com/qo9zk86IAH</a></p>&mdash; Christiaan Triebert (@trbrtc) <a href="https://twitter.com/trbrtc/status/869584650490040320">May 30, 2017</a></blockquote> <script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script> <p>My guess was a bit off as Christiaan's very nice breakdown shows, but I think I was close enough to call it a win. The lesson learned was to remember to always look at a place from all angles in Google Maps. Had I done that, I would have seen the yellow parasols hiding among the trees in the green area.</p> <p>Christiaan later challenged me with another picture which I also solved, but I won't go over how I did it, I will leave it as a challenge for you to find out. Maybe you can figure out how I did it. Let me know how you solved it. :) </p> <blockquote class="twitter-tweet tw-align-center" data-conversation="none" data-lang="en"><p lang="en" dir="ltr">If you feel like it, try this one. It is near (1-50km radius) of the other photo, to give a hint. <a href="https://t.co/Kutw2sK6wP">pic.twitter.com/Kutw2sK6wP</a></p>&mdash; Christiaan Triebert (@trbrtc) <a href="https://twitter.com/trbrtc/status/869587705251983361">May 30, 2017</a></blockquote> <script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script> <p>If you enjoyed this post, maybe you are also interested in reading my previous post about how I <a href="/blog/geolocating-maks-czuperski/">Geolocated Maks Czuperski</a>. If you would like to try out geolocation yourself, I would recommend keeping an eye on the <a href="https://twitter.com/search?f=tweets&vertical=default&q=%23DigitalSherlocks&src=typd">#DigitalSherlocks</a> hashtag on Twitter as people often tweet these sorts of challenges using that hashtag.</p> tag:michenriksen.com,2017-06-17:/blog/aquatone-tool-for-domain-flyovers/ AQUATONE: A tool for domain flyovers 2017-06-17T13:37:00Z 2017-06-17T13:37:00Z <%= render '/post_head.html', :post => item %> <div class="screenshot"> <a class="fancybox" href="/images/aquatone/u2.jpg" title="The Lockheed U-2 reconnaissance aircraft was given the codename Aquatone."> <img class="thumbnail" src="/images/aquatone/u2.jpg" alt="" /></a> <small>The Lockheed U-2 reconnaissance aircraft was given the codename Aquatone.</small> </div> <p>Knowing the attack surface of something is critical for both defending and attacking it. When it comes to domain names, a very common approach for uncovering the attack surface is to discover its subdomains. Subdomains will increase the number of potential target sites as well as uncover IP ranges to probe further.</p> <p>There are plenty of tools already for subdomain enumeration, e.g. <a href="http://tools.kali.org/information-gathering/fierce">Fierce</a>, <a href="https://github.com/TheRook/subbrute">SubBrute</a> and <a href="https://github.com/OJ/gobuster">Gobuster</a> however <a href="https://github.com/michenriksen/aquatone">AQUATONE</a> takes things a step further by not only doing classic brute force enumeration but also utilizing various open sources and internet services to dramatically increase the number of discovered subdomains. When subdomains have been discovered, AQUATONE can then be used to probe the hosts for common HTTP ports and gather response headers, HTML and screenshots to be compiled into a nice report for easy analysis.</p> <p>To make the tool as flexible as possible, AQUATONE is divided into three separate commands, so if you're only interested in using it for subdomain discovery without any scanning or screenshotting, you can easily do that. Lets go over the three phases of an AQUATONE assessment:</p> <h2>Phase 1: Discovery</h2> <p>To demonstrate the usage of AQUATONE, we will perform an assessment on the <code>corp.yahoo.com</code> domain. I have chosen this domain because Yahoo's <a href="https://hackerone.com/yahoo">Bug Bounty program</a> includes all of <code>*.yahoo.com</code> in their scope, so it should be acceptable to run a tool like AQUATONE against it.</p> <p>Kicking off the <code>aquatone-discover</code> tool:</p> <div class="screenshot"> <a class="fancybox" href="/images/aquatone/discover_start.png" title="Starting aquatone-discover against <code>corp.yahoo.com</code>..."> <img class="thumbnail" src="/images/aquatone/discover_start_thumbnail.png" alt="" /></a> <small>Starting aquatone-discover against <code>corp.yahoo.com</code>...</small> </div> <p>The first thing <code>aquatone-discover</code> does is to identify the authoritative name servers for the target domain. Using these name servers for resolution ensures that the information is up to date and discovery is maximised.</p> <p>It also does a quick test to see if the target domain is configured to be a wildcard domain as such domains can produce a lot of false positives. If the domain turns out to be a wildcard, it will identify the possible wildcard responses and filter them out. <code>corp.yahoo.com</code> is luckily not configured to be wildcard.</p> <p>After name server and wildcard detection, it proceeds to ask each subdomain collector module for potential subdomains under the target domain. <code>aquatone-discover</code> ships with following collector modules:</p> <p> <ul> <li>Dictionary brute force (see dictionary <a href="https://github.com/michenriksen/aquatone/blob/master/subdomains.lst">here</a>)</li> <li><a href="http://dnsdb.org/">DNSDB.org</a></li> <li><a href="https://www.google.com/transparencyreport/">Google Transparency Report</a></li> <li><a href="https://hackertarget.com/find-dns-host-records/">HackerTarget</a></li> <li><a href="http://searchdns.netcraft.com/">Netcraft</a></li> <li><a href="https://www.shodan.io/">Shodan</a> (requires API key)</li> <li><a href="https://www.threatcrowd.org/">ThreatCrowd</a></li> <li><a href="http://www.virustotal.com/">VirusTotal</a> (requires API key)</li> </ul> </p> <p>The collector modules returned a total of 12.282 potential subdomains that aquatone-discover attempts to resolve.</p> <div class="screenshot"> <a class="fancybox" href="/images/aquatone/discover_progress.png" title="aquatone-discover resolving subdomains. Hitting <code>Enter</code> will output a progress report."> <img class="thumbnail" src="/images/aquatone/discover_progress_thumbnail.png" alt="" /></a> <small>aquatone-discover resolving subdomains. Hitting <code>Enter</code> will output a progress report.</small> </div> <p>After a while, aquatone-discover has run through the list and uncovered a total of 1.958 live subdomains. It also analyzed the IPs and printed a list of potential IP subnet ranges which can be used for further probing:</p> <div class="screenshot"> <a class="fancybox" href="/images/aquatone/discover_finished.png" title="aquatone-discover uncovered a total of 1.958 live subdomains."> <img class="thumbnail" src="/images/aquatone/discover_finished_thumbnail.png" alt="" /></a> <small>aquatone-discover uncovered a total of 1.958 live subdomains.</small> </div> <p>It also wrote the discovered hosts to files in the aquatone assessment directory that is automatically created for the target domain. <code>hosts.txt</code> contains a comma-separated list of domains and their IP:</p> <pre> 224-si1.corp.yahoo.com, 224-si2.corp.yahoo.com, 227-si1.corp.yahoo.com, 227-si2.corp.yahoo.com, 232-si1.corp.yahoo.com, 232-si2.corp.yahoo.com, 351-si1.corp.yahoo.com, 351-si2.corp.yahoo.com, 998-dmz-foundry1.corp.yahoo.com, 998-dmz-foundry2.corp.yahoo.com, aa-dc1.wpe.stg.test.corp.yahoo.com, aa-dc2.wpe.stg.test.corp.yahoo.com, aaa1-1-a-gci.corp.yahoo.com, aaa1-2-a-gci.corp.yahoo.com, aahost1.stg.test.corp.yahoo.com, aahost2.stg.test.corp.yahoo.com, aahost3.stg.test.corp.yahoo.com, aahost4.stg.test.corp.yahoo.com, aape01.stg.test.corp.yahoo.com, aavm1.stg.test.corp.yahoo.com, ... </pre> <p>This file can be sliced and diced with common command line tools and loaded into other tools that you might use. <code>hosts.json</code> contains the same information in JSON format and is used by the other AQUATONE tools but can also be useful if you want to use the information with custom scripts.</p> <h2>Phase 2: Scanning</h2> <p>Having discovered a bunch of subdomains on <code>corp.yahoo.com</code> is already quite useful. We could stop here and start poking around with other tools or manual browsing, but lets instead make <code>aquatone-scan</code> do the hard work for us of finding which hosts might serve web content:</p> <div class="screenshot"> <a class="fancybox" href="/images/aquatone/scan.png" title="aquatone-scan finding open ports on hosts."> <img class="thumbnail" src="/images/aquatone/scan_thumbnail.png" alt="" /></a> <small>aquatone-scan finding open ports on hosts.</small> </div> <p><code>aquatone-scan</code> found a bunch of open HTTP ports across the different hosts. By default, it will scan the following TCP ports: 80, 443, 8000, 8080 and 8443 which are all very common ports for web services. You can of course change this to your own list of ports with the <code>--ports</code> option, or specify one of the built-in list aliases:</p> <p> <ul> <li><code>small</code>: 80, 443</li> <li><code>medium</code>: 80, 443, 8000, 8080, 8443 (same as default)</li> <li><code>large</code>: 80, 81, 443, 591, 2082, 2087, 2095, 2096, 3000, 8000, 8001, 8008, 8080, 8083, 8443, 8834, 8888</li> <li><code>huge</code>: 80, 81, 300, 443, 591, 593, 832, 981, 1010, 1311, 2082, 2087, 2095, 2096, 2480, 3000, 3128, 3333, 4243, 4567, 4711, 4712, 4993, 5000, 5104, 5108, 5800, 6543, 7000, 7396, 7474, 8000, 8001, 8008, 8014, 8042, 8069, 8080, 8081, 8088, 8090, 8091, 8118, 8123, 8172, 8222, 8243, 8280, 8281, 8333, 8443, 8500, 8834, 8880, 8888, 8983, 9000, 9043, 9060, 9080, 9090, 9091, 9200, 9443, 9800, 9981, 12443, 16080, 18091, 18092, 20720, 28017</li> </ul> </p> <p>Using a larger port list will of course let you discover more web services, but it will also increase the time it takes for <code>aquatone-scan</code> to finish.</p> <p><code>aquatone-scan</code> created two new files in the assessment directory for <code>corp.yahoo.com</code>: <code>open_ports.txt</code> is a simple comma-separated list of hosts and their open ports:</p> <pre>,443,443,443,443,443,443,443,443,443,443,443,443,443,443,443,443,443,443,443,80,80,80,443,443,80,443 </pre> <p><code>urls.txt</code> contains a list of URLs that can be used to request the web pages on the open ports:</p> <pre> http://bomgar.corp.yahoo.com/ http://bouncer.gh.corp.yahoo.com/ http://buzz.corp.yahoo.com/ http://cloud.corp.yahoo.com/ http://fifa.corp.yahoo.com/ http://gemini.corp.yahoo.com/ http://guest.corp.yahoo.com/ http://insights.corp.yahoo.com/ http://ipv6.corp.yahoo.com/ http://marketingcentral.corp.yahoo.com/ http://messenger.corp.yahoo.com/ http://request.corp.yahoo.com/ http://sas.corp.yahoo.com/ http://services.corp.yahoo.com/ http://shop.corp.yahoo.com/ http://si.corp.yahoo.com/ http://wireless.corp.yahoo.com/ https://bomgar.corp.yahoo.com/ https://bouncer.gh.corp.yahoo.com/ https://fast.corp.yahoo.com/ ... </pre> <p>These files are used for the next phase of the assessment but are also convenient for loading into other tools like <a href="https://github.com/ChrisTruncer/EyeWitness">EyeWitness</a> or slicing and dicing with <code>grep,</code> <code>cut,</code> <code>awk,</code> etc.</p> <h2>Phase 3: Gathering</h2> <p>We now know about subdomains and open ports on <code>*.corp.yahoo.com</code>, it's time to use <code>aquatone-gather</code> to collect HTTP responses and screenshots and compile it all into a nice report:</p> <div class="screenshot"> <a class="fancybox" href="/images/aquatone/gather_start.png" title="aquatone-gather crunching through the web pages."> <img class="thumbnail" src="/images/aquatone/gather_start_thumbnail.png" alt="" /></a> <small>aquatone-gather crunching through the web pages.</small> </div> <p><code>aquatone-gather</code> loaded data from the files created by the previous AQUATONE tools and started requesting URLs to collect HTTP responses and screenshots. Behind the scenes, it uses <a href="https://github.com/segmentio/nightmare">Nightmare</a> for all the heavy lifting of requesting and screenshotting.</p> <p>Unfortunately Nightmare, and any other browser automation tool, is a bit flaky and will fail on some of the page processings as can be seen in the screenshot. I think the failure rate is acceptable, but something to be aware of. </p> <p>After a little while, it finishes processing all the web pages:</p> <div class="screenshot"> <a class="fancybox" href="/images/aquatone/gather_finish.png" title="aquatone-gather finished processing web pages."> <img class="thumbnail" src="/images/aquatone/gather_finish_thumbnail.png" alt="" /></a> <small>aquatone-gather finished processing web pages.</small> </div> <p>It prints a short summary of successful vs. failed page processings and a list of generated report pages, but this is far from the only files that <code>aquatone-gather</code> generated. Navigating to the assessment folder, we can see three new folders: <code>headers</code>, <code>html</code>, <code>report</code> and <code>screenshots</code>.</p> <p>The <code>headers</code> folder contains text files with response headers from all the page visits:</p> <pre> <strong>root@kali:~/aquatone/corp.yahoo.com/headers# cat bomgar_corp_yahoo_com__98_136_205_152__443.txt</strong> Cache-Control: no-cache Connection: Keep-Alive Content-Type: text/html; charset=utf-8 Date: Wed, 14 Jun 2017 12:22:01 GMT Expires: Thu, 19 Nov 1981 08:52:00 GMT Keep-Alive: timeout=15, max=100 Pragma: no-cache Server: Bomgar Set-Cookie: ns_s=c9b9309296cf5babeb7e193125cb2cf0f3c7f13c; path=/; secure; HttpOnly Strict-Transport-Security: max-age=31536000 Transfer-Encoding: chunked X-Ua-Compatible: IE=edge <strong>root@kali:~/aquatone/corp.yahoo.com/headers#</strong> </pre> <p>These files can be very useful with <code>grep</code> and other tools to quickly find information on server technology and other things that are interesting from a security point of view.</p> <p>The <code>html</code> folder contains HTML bodies from all the page visits:</p> <pre> <strong>root@kali:~/aquatone/corp.yahoo.com/html# cat bomgar_corp_yahoo_com__98_136_205_152__443.html</strong> &lt;!DOCTYPE html PUBLIC &quot;-//W3C//DTD XHTML 1.1//EN&quot; &quot;http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd&quot;&gt; &lt;html xmlns=&quot;http://www.w3.org/1999/xhtml&quot; xml:lang=&quot;en-us&quot;&gt; &lt;head&gt; &lt;meta http-equiv=&quot;Content-Type&quot; content=&quot;text/html; charset=utf-8&quot; /&gt; &lt;title&gt;Yahoo! Global Service Desk LiveChat&lt;/title&gt; &lt;meta name=&quot;viewport&quot; content=&quot;width=device-width, initial-scale=1.0, maximum-scale=1.0&quot; /&gt; &lt;link href=&quot;/content/common.css&quot; rel=&quot;stylesheet&quot; type=&quot;text/css&quot; /&gt; &lt;link href=&quot;/content/public.css&quot; rel=&quot;stylesheet&quot; type=&quot;text/css&quot; /&gt; &lt;link href=&quot;/content/mobile.css&quot; rel=&quot;stylesheet&quot; type=&quot;text/css&quot; /&gt; &lt;/head&gt; &lt;body&gt; &lt;div id=&quot;container&quot;&gt; &lt;div id=&quot;header&quot; class=&quot;contentBox&quot;&gt; ... &lt;div style=&quot;display: none&quot;&gt; &lt;div style=&quot;margin: 1em;&quot;&gt; &lt;a href=&quot;http://www.bomgar.com&quot; class=&quot;inverse&quot; target=&quot;_blank&quot;&gt;Secure Remote Desktop Access by Bomgar&lt;/a&gt; &lt;/div&gt; &lt;/div&gt; &lt;/div&gt; &lt;/body&gt; &lt;/html&gt; <strong>root@kali:~/aquatone/corp.yahoo.com/html#</strong> </pre> <p>There are tons of things that these files can be used for. More on this later.</p> <p>The <code>screenshots</code> folder contains, as the name might suggest, PNG screenshots of all the page visits:</p> <pre> <strong>root@kali:~/aquatone/corp.yahoo.com/screenshots# ls</strong> bomgar_corp_yahoo_com__98_136_205_152__443.png bomgar_corp_yahoo_com__98_136_205_152__80.png bouncer_gh_corp_yahoo_com__72_30_2_113__443.png bouncer_gh_corp_yahoo_com__72_30_2_113__80.png buzz_corp_yahoo_com__77_238_184_150__80.png cloud_corp_yahoo_com__77_238_184_150__80.png ... si_corp_yahoo_com__77_238_184_150__80.png vpn1-1-gci_eglbp_corp_yahoo_com__203_83_249_4__443.png vpn1-1-ptn_corp_yahoo_com__216_145_48_151__443.png vpn1-1-ptn_eglbp_corp_yahoo_com__203_83_249_10__443.png vpn1-2-gci_sv6_corp_yahoo_com__209_131_62_228__443.png vpn-1-gci_hongkong_corp_yahoo_com__117_104_189_54__443.png vpn2-1-gci_eglbp_corp_yahoo_com__203_83_249_5__443.png vpn2-1-ptn_corp_yahoo_com__216_145_48_152__443.png vpn2-2-gci_sv6_corp_yahoo_com__209_131_62_229__443.png vpn-2-gci_sv6_corp_yahoo_com__209_131_62_230__443.png wireless_corp_yahoo_com__77_238_184_150__80.png <strong>root@kali:~/aquatone/corp.yahoo.com/screenshots#</strong> </pre> <p>You can of course browse these screenshots directly in the folder, but it's probably more useful to analyse them by opening the generated HTML report page:</p> <div class="screenshot"> <a class="fancybox" href="/images/aquatone/aquatone_report.gif" title="Browsing the AQUATONE report."> <img class="thumbnail" src="/images/aquatone/aquatone_report_thumbnail.png" alt="" /></a> <small>Browsing the AQUATONE report (click to see animated Gif).</small> </div> <p>The report lines up the screenshots with response headers so that you quickly scan through the collected information for interesting pages. AQUATONE will highlight headers that may increase security with a green background and headers that may present a security issue with a red background. Before you go on a bug bounty spree with this, please remember that god strangles a puppy every time someone reports missing <code>X-Frame-Options</code>. ;)</p> <h2>CLI tricks</h2> <p>The generated report is the final product of AQUATONE, but lots of useful stuff can be done with all the raw files that are generated in the assessment folder, so let's wrap up this blog post with some examples of what you can do:</p> <h3>Get server technology stats</h3> <pre> <strong>root@kali:~/aquatone/corp.yahoo.com/headers# cat * | grep 'Server:' | sort | uniq -c | sort -nr</strong> 13 Server: ATS 6 Server: Bomgar 1 Server: AkamaiGHost <strong>root@kali:~/aquatone/corp.yahoo.com/headers#</strong> </pre> <h3>Find more subdomains</h3> <pre> <strong>root@kali:~/aquatone/corp.yahoo.com/html# cat * | egrep -o '[a-z0-9\-\_\.]+\.corp\.yahoo\.com' | sort -u</strong> bomgar.corp.yahoo.com bouncer.by.corp.yahoo.com fast.corp.yahoo.com it.corp.yahoo.com request.corp.yahoo.com services.corp.yahoo.com <strong>root@kali:~/aquatone/corp.yahoo.com/html#</strong> </pre> <h3>Find HTML comments</h3> <pre> <strong>root@kali:~/aquatone/corp.yahoo.com/html# cat * | egrep -o '&lt;!--.*--&gt;'</strong> &lt;!--//--&gt; &lt;!-- Begin comScore Tag --&gt; &lt;!-- bouncer02.gh.bf1.yahoo.com Wed Jun 14 12:22:09 UTC 2017 --&gt; &lt;!-- bouncer12-os.gh.bf2.yahoo.com Wed Jun 14 12:22:29 UTC 2017 --&gt; &lt;!-- #doc4 --&gt; &lt;!-- .dw1 --&gt; &lt;!-- .dw4 --&gt; ... &lt;!-- /.shmod --&gt; &lt;!-- SpaceID=0 timeout (ads1) --&gt; &lt;!-- src2.ops.ir2.yahoo.com Wed Jun 14 12:22:15 UTC 2017 --&gt; &lt;!-- src4.ops.ir2.yahoo.com Wed Jun 14 12:21:44 UTC 2017 --&gt; &lt;!-- src4.ops.ir2.yahoo.com Wed Jun 14 12:21:51 UTC 2017 --&gt; &lt;!-- src4.ops.ir2.yahoo.com Wed Jun 14 12:22:27 UTC 2017 --&gt; &lt;!-- src6.ops.ir2.yahoo.com Wed Jun 14 12:21:57 UTC 2017 --&gt; &lt;!-- src6.ops.ir2.yahoo.com Wed Jun 14 12:22:15 UTC 2017 --&gt; &lt;!-- src6.ops.ir2.yahoo.com Wed Jun 14 12:22:36 UTC 2017 --&gt; &lt;!-- URL: /::ProfilerTotal:557:1497442917838::Page Creation:40:1497442917838::user_ups:0:1497442917844::ydht_time:1:1497442917845::Maple Execution:518:1497442917878::Maple WS:41:1497442917879::SHAdModule:457:1497442917921::SHLeftNavigationModule:7:1497442918378::SHHeroModule:0:1497442918385::SHBrowseShoppingModule:5:1497442918385::SHSocialNewBrowseModule:0:1497442918390::SHCopyrightModule:1:1497442918391:: --&gt; &lt;!-- web23.shop.bf1.yahoo.com --&gt; &lt;!-- web23.shop.bf1.yahoo.com Wed Jun 14 12:21:57 UTC 2017 --&gt; </pre> <h3>Find pages with password fields</h3> <pre> <strong>root@kali:~/aquatone/corp.yahoo.com/html# grep 'type="password"' *</strong> bouncer_gh_corp_yahoo_com__72_30_2_113__80.html: &lt;dd&gt;&lt;input class=&quot;input-large&quot; name=&quot;pass_word&quot; type=&quot;password&quot; id=&quot;pass_word&quot; maxlength=&quot;64&quot; autocomplete=&quot;off&quot; autocorrect=&quot;off&quot; autocapitalize=&quot;off&quot; spellcheck=&quot;false&quot; &gt;&lt;/dd&gt; fast_corp_yahoo_com__98_136_205_216__443.html: &lt;dd&gt;&lt;input class=&quot;input-large&quot; name=&quot;pass_word&quot; type=&quot;password&quot; id=&quot;pass_word&quot; maxlength=&quot;64&quot; autocomplete=&quot;off&quot; autocorrect=&quot;off&quot; autocapitalize=&quot;off&quot; spellcheck=&quot;false&quot; &gt;&lt;/dd&gt; <strong>root@kali:~/aquatone/corp.yahoo.com/html#</strong> </pre> <h3>Get hosts listening on port 443</h3> <pre> <strong>root@kali:~/aquatone/corp.yahoo.com# cat open_ports.txt | grep ',443' | cut -d "," -f 1</strong> ... <strong>root@kali:~/aquatone/corp.yahoo.com#</strong> </pre> <h3>Check HTTPS hosts for Heartbleed</h3> <pre> <strong>root@kali:~/aquatone/corp.yahoo.com# grep https urls.txt | cut -d '/' -f 3 > /tmp/targets.lst</strong> <strong>root@kali:~/aquatone/corp.yahoo.com# sslscan --targets=/tmp/targets.lst --no-ciphersuites --no-fallback --no-renegotiation --no-compression --no-check-certificate</strong> Version: 1.11.9-static OpenSSL 1.0.2l-dev xx XXX xxxx Testing SSL server bomgar.corp.yahoo.com on port 443 using SNI name Heartbleed: TLS 1.2 not vulnerable to heartbleed TLS 1.1 not vulnerable to heartbleed TLS 1.0 not vulnerable to heartbleed Testing SSL server bouncer.gh.corp.yahoo.com on port 443 using SNI name ... Testing SSL server vpn2-2-gci.sv6.corp.yahoo.com on port 443 using SNI name Heartbleed: TLS 1.2 not vulnerable to heartbleed TLS 1.1 not vulnerable to heartbleed TLS 1.0 not vulnerable to heartbleed <strong>root@kali:~/aquatone/corp.yahoo.com#</strong> </pre> <p>That's it! I hope you will take AQUATONE on a test flight and let me know what you think. You can find installation instructions in the project <a href="https://github.com/michenriksen/aquatone#installation">README</a>.</p> tag:michenriksen.com,2017-01-18:/blog/geolocating-maks-czuperski/ Geolocating Maks Czuperski 2017-01-18T13:37:00Z 2017-01-18T13:37:00Z <%= render '/post_head.html', :post => item %> <p>The other day while checking my Twitter feed, I came across an interesting retweet from <a href="https://twitter.com/EliotHiggins">Elliot Higgins</a>, the founder of <a href="https://www.bellingcat.com/">Bellingcat</a>:</p> <blockquote class="twitter-tweet tw-align-center" data-lang="en"><p lang="en" dir="ltr"><a href="https://twitter.com/hashtag/DigitalSherlocks?src=hash">#DigitalSherlocks</a> out there!—Where am I? <a href="https://t.co/4eXw7BuMj8">pic.twitter.com/4eXw7BuMj8</a></p>&mdash; Maks Czuperski (@MaksCzuperski) <a href="https://twitter.com/MaksCzuperski/status/819510013148463105">January 12, 2017</a></blockquote> <script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script> <p>This looked like a fun challenge and being the OSINT geek that I am, I couldn't let it pass. Here is how I geolocated Maks Czuperski:</p> <p>First thing was of course to get a copy of the image. Right-click and <em>Copy Image Address</em> gave me a good 2048x1536px version where details could be picked out.</p> <div class="screenshot"> <a class="fancybox" href="/images/geolocation/whereami.png" title="The picture from Maks Czuperski's tweet"> <img class="thumbnail" src="/images/geolocation/whereami_thumbnail.png" alt="" /></a> <small>The picture from Maks Czuperski's tweet (<a href="/images/geolocation/whereami_large.png" target="_blank">click here</a> for high resolution)</small> </div> <p>The location did not look familiar to me and there were no famous buildings in the background, readable street signs or other easy wins to quickly figure out the location. The building across the plaza did have a kind of interesting shape and colors, so the first thing I tried was to cut out a piece of the picture with part of the building clearly visible without obstructions, and upload it to Google's reverse image search. It is quite impressive, and almost eerie, what Google can recognize in images. This time however, I had no luck. I also tried with a crop of the walkway pattern in the lower area of the picture, but no luck there either.</p> <p>Next step was to zoom in on the image and comb it for any details that could give me a hint to its location. This was when I spotted the store signs in the lower right corner of the image:</p> <div class="screenshot"> <a class="fancybox" href="/images/geolocation/whereami_zoom.png" title="An Expert store is visible in the image"> <img class="thumbnail" src="/images/geolocation/whereami_zoom_thumbnail.png" alt="" /></a> <small>An Expert store is visible in the image</small> </div> <p>I was lucky to know that Expert is an electronics store from my time living in Denmark and, assuming that Expert only has stores in Denmark, I wondered if it could be somewhere in Copenhagen perhaps. A quick Google search led me to the company's <a href="https://en.wikipedia.org/wiki/Expert_(company)">Wikipedia article</a> which stated that Expert operates in several countries a part from Denmark:</p> <p> <ul> <li>Norway</li> <li>Denmark</li> <li>Sweden</li> <li>Faroe Islands</li> <li>Åland Islands</li> <li>Finland</li> <li>Estonia</li> <li>Belgium</li> <li>Germany</li> <li>Greece</li> <li>Ireland</li> <li>France</li> <li>Austria</li> <li>Czech Republic</li> <li>Iceland</li> <li>Italy</li> <li>The Netherlands</li> <li>Portugal</li> <li>Spain</li> <li>Slovakia</li> <li>Australia/New Zealand</li> <li>Croatia</li> <li>Hungary</li> </ul> </p> <p>This made the list of potential places quite a bit bigger, but at least it was narrowed down to a handful of countries. I combed through the picture once more to find any other details I could pair with this information and pretty quickly spotted a sign for a place called "Gold Mountain" right next to the Expert shop:</p> <div class="screenshot"> <a class="fancybox" href="/images/geolocation/whereami_gold_mountain.png" title="A place called Gold Mountain next to Expert"> <img class="thumbnail" src="/images/geolocation/whereami_gold_mountain_thumbnail.png" alt="" /></a> <small>A place called Gold Mountain next to Expert</small> </div> <p>I had never heard of Gold Mountain before, and it was my hope that this would be a bit more unique than Expert. I made the assumption that he was most likely in a capital city, or at least a bigger city, in one of the countries where Expert operates. My plan was now to perform a search for <em>Gold Mountain</em> combined with the capital city name for each country in the list:</p> <div class="screenshot"> <a class="fancybox" href="/images/geolocation/google_gold_mountain.png" title="Googling for <em>Gold Mountain</em> in Oslo"> <img class="thumbnail" src="/images/geolocation/google_gold_mountain_thumbnail.png" alt="" /></a> <small>Googling for <em>Gold Mountain</em> in Oslo</small> </div> <p>Bingo! I was lucky that the capital for the first country in the list yielded a result that looked like the right place. The Google Streetview image to the right of the search results looked like it had the funnily shaped building in the background, and sure enough, clicking on it <a href="https://encrypted.google.com/maps/uv?hl=en&pb=!1s0x46416e87584d4a15:0xec11d10f5cae7d6c!2m19!2m2!1i80!2i80!3m1!2i20!16m13!1b1!2m2!1m1!1e1!2m2!1m1!1e3!2m2!1m1!1e5!2m2!1m1!1e4!3m1!7e115!4s/maps/place/%2522golden%2Bmountain%2522%2Boslo/@59.9130538,10.7342323,3a,75y,315.79h,90t/data%3D*213m4*211e1*213m2*211sg-lA7LDc3hCsqgBfS2QtPQ*212e0*214m2*213m1*211s0x0:0xec11d10f5cae7d6c?hl%3Den!5s%22golden+mountain%22+oslo+-+Google+Search&imagekey=!1e2!2sg-lA7LDc3hCsqgBfS2QtPQ&sa=X&ved=0ahUKEwjzu6bc8MvRAhXGAxoKHTyXDIUQpx8IbjAK">landed me</a> straight on the plaza in the picture!</p> <p>I pretty quickly determined the direction that the picture was taken from and tweeted at Maks Czuperski:</p> <blockquote class="twitter-tweet tw-align-center" data-lang="en"><p lang="en" dir="ltr"><a href="https://twitter.com/MaksCzuperski">@MaksCzuperski</a> <a href="https://twitter.com/EliotHiggins">@EliotHiggins</a> Somewhere here? <a href="https://t.co/FplgM8BPop">pic.twitter.com/FplgM8BPop</a></p>&mdash; Michael Henriksen (@michenriksen) <a href="https://twitter.com/michenriksen/status/819517437259476992">January 12, 2017</a></blockquote> <script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script> <p>After a while he replied and asked me to be more precise. I tried to narrow it down further:</p> <blockquote class="twitter-tweet tw-align-center" data-lang="en"><p lang="en" dir="ltr"><a href="https://twitter.com/MaksCzuperski">@MaksCzuperski</a> <a href="https://twitter.com/EliotHiggins">@EliotHiggins</a> My guess would be one of these top windows. It&#39;s Fridtjof Nansens Plass in Oslo. <a href="https://t.co/y1TmuzmoyU">pic.twitter.com/y1TmuzmoyU</a></p>&mdash; Michael Henriksen (@michenriksen) <a href="https://twitter.com/michenriksen/status/819527342745878528">January 12, 2017</a></blockquote> <script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script> <p>I managed to narrow it down to a couple of windows where the picture could have been taken from, which I think was pretty cool. It was a super fun exercise and I wish someone would arrange daily or weekly geolocation challenges like this!</p> <p>If you find this sort of stuff interesting, you should definitely check out <a href="https://www.bellingcat.com/">Bellingcat</a>'s articles and guides. In case you haven't heard of them, here's a short description of what they are about, from their own website:</p> <blockquote> Bellingcat uses open source and social media investigation to investigate a variety of subjects, from Mexican drug lords to conflicts being fought across the world. Bellingcat brings together contributors who specialise in open source and social media investigation, and creates guides and case studies so others may learn to do the same. </blockquote> <p>As a little bonus to my investigation, I later came across the following tweet:</p> <blockquote class="twitter-tweet tw-align-center" data-lang="en"><p lang="en" dir="ltr">Thanks, <a href="https://twitter.com/MaksCzuperski">@MaksCzuperski</a> for more insight on social media and the engagement age. <a href="https://twitter.com/Atlantkomite">@Atlantkomite</a> <a href="https://t.co/9eTyvf0NSN">pic.twitter.com/9eTyvf0NSN</a></p>&mdash; Marita I. Wangberg (@FD_Marita) <a href="https://twitter.com/FD_Marita/status/819497169413607424">January 12, 2017</a></blockquote> <script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script> <p>The picture shows Maks Czuperski doing a presentation on geolocation stuff at the Norwegian Atlantic Committee. He apparently used his geolocation challenge tweet as part of his presentation, to demonstrate how quickly someone could find him. Pretty cool!</p> tag:michenriksen.com,2016-11-14:/blog/detecting-online-investigations/ Detecting online investigations 2016-11-14T13:37:00Z 2016-11-14T13:37:00Z <%= render '/post_head.html', :post => item %> <p>I recently purchased a license for <a href="https://www.hunch.ly/">Hunchly</a>, an awesome tool for performing online investigations and general research on the web. The tool seems to be very popular among criminal investigators, journalists and OSINT geeks like me.</p> <p>For those who don't know, Hunchly is a tool that integrates directly into your browser through an extension to record and store local copies of every website you visit during an investigation. This is very convenient when doing OSINT investigations as you never loose anything, even if it gets deleted at a later time, and it makes it easy to trace your steps to how you found a particular piece of information.</p> <p>Another great feature of Hunchly is the ability to associate so-called <em>selectors</em> to cases. Selectors are basically specific strings of interest that Hunchly will then keep track of, and notify you whenever these selectors are found on websites you visit. Selectors would normally be things like names, email addresses, phone numbers, domains and anything else that is relevant to the person or subject you are investigating.</p> <p>Hunchly can be configured to automatically highlight these selectors on websites when they are encountered. It is not enabled by default, but I'm sure it's a very common setting to enable as it makes it very easy to spot the relevant parts when browsing:</p> <div class="screenshot"> <a class="fancybox" href="/images/hunchly/hunchly_selectors.png" title="Highlighted selectors on the michenriksen.com frontpage."> <img class="thumbnail" src="/images/hunchly/hunchly_selectors_thumbnail.png" alt="" /></a> <small>Highlighted selectors on the michenriksen.com frontpage.</small> </div> <p>The above screenshot shows how my website's frontpage looks like when Hunchly is configured with the selectors: <em>Michael Henriksen</em>, <em>SoundCloud</em> and <em>Gitrob</em>. All occurrences of these strings are highlighted with a yellow background.</p> <p>Since Hunchly is highlighting the selectors by modifying the HTML before it's displayed in the browser, I started thinking about how a tech-savvy criminal, or anyone doing activities that would cause people to investigate them, such as an activist, could exploit this feature to detect the fact that someone is actively investigating them, and even get a sense of what the investigator knows and what they are interested in.</p> <p>This is how Hunchly highlights selector strings:</p> <pre> &lt;mark data-markjs=&quot;true&quot; class=&quot;hunchly&quot;&gt;selector text&lt;/mark&gt; </pre> <p>Hunchly wraps the selectors in <code>&lt;mark&gt;</code> tags with a unique class of <code>hunchly</code> which can be used to very accurately determine that Hunchly is being used on a website.</p> <p>Using a bit of Javascript, it is fairly easy for anyone worried about being investigated to detect it. Here is a small Proof Of Concept:</p> <script src="https://gist.github.com/michenriksen/3ace5b2714097d05ea18bc843b1719ad.js"></script> <p>On <strong>line 9 - 11</strong> a <code>&lt;div&gt;</code> element is defined with an ID of <code>canary</code>. This element would contain potential selectors that an investigator is likely to have entered into Hunchly. This should obviously not contain actual sensitive information that would help an investigator, but already publicly known information such as a name, public email address, etc. which is very likely for an investigator to have entered into Hunchly. It could also contain misinformation (e.g. a fake phone number) that has been planted on other websites. This would give the person under investigation an idea of where the investigator has already been, and what the investigator thinks they know about their target.</p> <p>On <strong>line 14</strong> the Javascript is looking for any <code>&lt;mark&gt;</code> tags with a <code>hunchly</code> class within the <code>#canary</code> div element. If no elements are found, it simply logs <em>No Hunchly selectors found in canary</em> to the console.</p> <p><strong>Line 18 - 22</strong> is what is being executed if a Hunchly selector highlight is found within the <code>#canary</code> element. It logs the total amount of selectors detected and then loops over each selector to log the text that was highlighted. It finishes off with popping up an alert box with <em>Hi there, investigator!</em> as the message.</p> <p>Instead of simply logging the information to the console, the script could easily be modified to call out to a backend script with Ajax or do something more aggressive like triggering a browser exploit, loading a <a href="http://beefproject.com/">BeEF</a> hook, redirecting them to Goatse or anything else to mess with the investigator. I'm sure you can come up with other fun things to do...</p> <p>Before I finish this post, I want emphasize that this is not meant as a diss to Hunchly, I think it's an excellent tool and would recommend it to anyone doing OSINT stuff. I simply felt like pointing out a potential problem with Hunchly's selector highlight feature, so if you are looking into someone who might be crazy enough to do stuff like this, you might want to turn that feature off. I don't know if this is the reason it is disabled as default, or if there is another reason. Anyways, check out <a href="https://www.hunch.ly">Hunchly</a> website or watch this <a href="https://www.youtube.com/watch?v=wA1ec0dPYhw">excellent video tutorial</a> if you want to know more about the tool.</p> tag:michenriksen.com,2016-10-14:/blog/birdwatcher-twitter-osint-framework/ Birdwatcher: Data analysis and OSINT framework for Twitter 2016-10-14T13:37:00Z 2016-10-14T13:37:00Z <%= render '/post_head.html', :post => item %> <p>Yes, here again with another tool release. This time it's an <a href="https://en.wikipedia.org/wiki/Open-source_intelligence">OSINT</a> framework for a different social network &mdash; <a href="https://twitter.com/">Twitter.com</a>.</p> <div class="screenshot"> <a class="fancybox" href="/images/birdwatcher/start.png" title="Birdwatcher framework"> <img class="thumbnail" src="/images/birdwatcher/start_thumbnail.png" alt="" /></a> <small>Birdwatcher framework</small> </div> <p><a href="https://github.com/michenriksen/birdwatcher">Birdwatcher</a> started out as a collection of small scripts to generate a classic weighted word cloud of Tweets from a group of users. As I thought about what else I could do with data from Twitter I decided to rewrite the scripts into a full-fledged, module based, console framework with a ton more functionality.</p> <p>If you have any experience working with other frameworks such as <a href="https://www.metasploit.com/">Metasploit</a> or <a href="https://bitbucket.org/LaNMaSteR53/recon-ng">Recon-ng</a>, you will feel right at home with Birdwatcher as it's heavily inspired by these frameworks and has many of the same concepts and commands.</a></p> <p>This blog post won't go over how to set up Birdwatcher, but you can have a look at the <a href="https://github.com/michenriksen/birdwatcher#birdwatcher">README</a> to find out how to <a href="https://github.com/michenriksen/birdwatcher#installation">install</a> and <a href="https://github.com/michenriksen/birdwatcher#configuration">configure</a> the framework.</p> <h2>Workspaces</h2> <p>Just like Metasploit and Recon-ng, Birdwatcher supports the concept of Workspaces. Workspaces enable you to segment and manage users and data stored in the underlying database. You can use workspaces to create logical separation between different users. For example, you may want to create a workspace for a company, a department or for a specific topic.</p> <p>The command prompt will always show the currently active workspace inside the square brackets. Birdwatcher will always have a default workspace which might be all you need if you intend to use Birdwatcher on a single group of users. If you plan to use it on several different groups, it is recommended to create a workspace for each of them, to prevent cross contamination.</p> <h2>Commands</h2> <p>The core of the Birdwatcher framework is its commands and one of the most important ones is the <code>help</code> command:</p> <div class="screenshot"> <a class="fancybox" href="/images/birdwatcher/help.png" title="Executing the help command"> <img class="thumbnail" src="/images/birdwatcher/help_thumbnail.png" alt="" /></a> <small>Executing the help command</small> </div> <p>The <code>help</code> command simply lists all available commands with short descriptions of what they do.</p> <h2>Modules</h2> <p>Again, just like Metasploit and Recon-ng, Birdwatcher ships with a bunch of modules that either enrich the raw Twitter data harvested by the commands or somehow present the data in interesting and useful ways. Here are some of the things the modules can currently do:</p> <ul> <li>Retrieve user's Klout score, Tweet topics and influence graph</li> <li>Generate weighted word clouds based on user's Tweets</li> <li>Listing the most shared URLs</li> <li>Generate graphical social graphs between users</li> <li>Crawl shared URLs to retrieve HTTP status codes, content types and page titles</li> <li>Generate KML files with geo-enabled Tweets to be viewed in Google Earth</li> <li>Generate Punchcard-style plots of when users are most engaged with Twitter</li> <li>Calculate the sentiment score of Tweets (positive, neutral or negative)</li> </ul> <p>Birdwatcher's code is designed to make it pretty simple for anyone with a bit of Ruby knowledge to extend Birdwatcher with new modules. How to create one is out of scope for this blog post, but have a look at this <a href="https://github.com/michenriksen/birdwatcher/wiki/Creating-a-Birdwatcher-Module">Wiki article</a> if you are interested in finding out more.</p> <h2>LOVELY HORSE</h2> <p>If you have been following the news around the Snowden documents, you might have heard of a program by the UK intelligence agency GCHQ called <a href="http://www.mirror.co.uk/news/technology-science/technology/gchqs-lovely-horse-tool-helped-5133474">LOVELY HORSE</a>. The program was made to simply monitor a smaller group of security related Twitter accounts to keep taps on what was being said and possibly more.</p> <p>To demonstrate the capabilities and usage of Birdwatcher, I thought it would be fun to go through how we can create our own LOVELY HORSE program...</p> <h3>Creating a new workspace</h3> <p>Instead of using the default workspace, let's create a dedicated one for our lovely horses to keep things neat and tidy:</p> <div class="screenshot"> <a class="fancybox" href="/images/birdwatcher/workspace.png" title="Creating a new workspace"> <img class="thumbnail" src="/images/birdwatcher/workspace_thumbnail.png" alt="" /></a> <small>Creating a new workspace</small> </div> <p>The <code>workspace add</code> command created our new workspace and automatically made it the currently active one, as can be seen in the square brackets of the command prompt.</p> <h3>Adding users to the workspace</h3> <p>Now that we have our workspace we need to add some users to it so we have something to work with. The <a href="https://s3.amazonaws.com/s3.documentcloud.org/documents/1588722/lovely-horse.pdf">leaked PDF</a> contains a list of 37 Twitter accounts that we will use for this example:</p> <pre> 0xcharlie alexsotirov anon_central anon_operations anonops anonymousirc bradarkin CeRTFi danchodanchev daveaitel dinodaizovi diocyde egyp7 GoVCeRT_NL halvarflake hdmoore hernano JaNeTCSiRT kevinmitnick lennyzeltser lulzsec mdowd mikko msftsecresponse operationleaks owasp pusscat Shadowserver snowfl0w taosecurity taviso teamcymru thegrugq TheHackersNews tinman2k VuPeN WTFuzz </pre> <p>One way to add the users would be to execute <code>user add 0xcharlie alexsotirov ... WTFuzz</code> but that would be a lot of typing and I don't really like that. Instead we can make use of our first module to easily import them into the workspace. We copy the usernames and save them to a file and load the User Importer module:</p> <div class="screenshot"> <a class="fancybox" href="/images/birdwatcher/module_info.png" title="The User Importer module"> <img class="thumbnail" src="/images/birdwatcher/module_info_thumbnail.png" alt="" /></a> <small>The User Importer module</small> </div> <p>The <code>use</code> command loads a module by its <em>path</em>. The path is determined simply by how the module files are placed in the directory stucture. Modules live inside at least one directory which can be seen as a namespace of the type of object they are working on. In this case the User Importer lives in the <code>users/</code> namespace which makes pretty good sense. When a module is loaded it is also indicated in the command prompt with another set of square brackets with the module's path in red text.</p> <p>After loading the module we type <code>show info</code> to get a bit more information on what the module does. All modules have additional information that can be seen with the <code>show info</code> command.</p> <p>The <code>show</code> command can also display any options a module might have:</p> <div class="screenshot"> <a class="fancybox" href="/images/birdwatcher/module_options.png" title="Options for the User Importer module"> <img class="thumbnail" src="/images/birdwatcher/module_options_thumbnail.png" alt="" /></a> <small>Options for the User Importer module</small> </div> <p>The module is very basic and only has one option called <code>FILE</code> which tells the module which file to read usernames from. The table tells us that the option is required to set and that the current value is empty. Let's configure the module and run it:</p> <div class="screenshot"> <a class="fancybox" href="/images/birdwatcher/user_import.png" title="Importing users"> <img class="thumbnail" src="/images/birdwatcher/user_import_thumbnail.png" alt="" /></a> <small>Importing users</small> </div> <p>The module fetched basic user information from the Twitter API and saved them to the underlying database. We can see the users in the current workspace at any time with the <code>user list</code> command:</p> <div class="screenshot"> <a class="fancybox" href="/images/birdwatcher/user_list.png" title="Paging through users in the workspace"> <img class="thumbnail" src="/images/birdwatcher/user_list_thumbnail.png" alt="" /></a> <small>Paging through users in the workspace</small> </div> <h3>Fetching Tweets</h3> <p>Now that we have imported our lovely horses we can fetch their Tweets from the Twitter API and have them saved to the database for analysis:</p> <div class="screenshot"> <a class="fancybox" href="/images/birdwatcher/status_fetch.png" title="Fetching Tweets from users"> <img class="thumbnail" src="/images/birdwatcher/status_fetch_thumbnail.png" alt="" /></a> <small>Fetching Tweets from users</small> </div> <p>The <code>status fetch</code> command will fetch up to 1.000 Tweets from each user and save them to the database. The command also extracts entities such as URLs, Mentions and Hashtags to save them to separate database tables. The command might take a bit of time to finish the first time because of all the Tweets it needs to fetch and process, however on any subsequent runs, it only fetches and processes any Tweets the users might have posted since the last run.</p> <p>Now that we have fetched the Tweets we can page through them with the <code>status list</code> command:</p> <div class="screenshot"> <a class="fancybox" href="/images/birdwatcher/status_list.png" title="Listing Tweets from users"> <img class="thumbnail" src="/images/birdwatcher/status_list_thumbnail.png" alt="" /></a> <small>Listing Tweets from users</small> </div> <p>Using the <code>status search</code> command we can find Tweets containing a specific word or phrase, for example <em>lovelyhorse</em>:</p> <div class="screenshot"> <a class="fancybox" href="/images/birdwatcher/status_search.png" title="Searching for Tweets mentioning <em>lovelyhorse</em>"> <img class="thumbnail" src="/images/birdwatcher/status_search_thumbnail.png" alt="" /></a> <small>Searching for Tweets mentioning <em>lovelyhorse</em></small> </div> <h3>Crawling URLs</h3> <p>After the Tweets have been fetched and processed we also have a pretty large collection of URLs that might point to interesting or valuable information. Right now we only know the URLs that were shared which can pretty hard to process. To get a better idea of which links might be interesting we can use the URL Crawler module:</p> <div class="screenshot"> <a class="fancybox" href="/images/birdwatcher/url_crawl_info.png" title="Loading the URL Crawler module"> <img class="thumbnail" src="/images/birdwatcher/url_crawl_info_thumbnail.png" alt="" /></a> <small>Loading the URL Crawler module</small> </div> <p>As the module information says, it enriches the collected URLs with their HTTP status codes, content types and potentially page titles if the URL points to a HTML page with a title. The module also follows redirects so in case the URL is somehow obfuscated or shortened we can know the actual destination too.</p> <p>The module also warns us that it might not be safe to blindly visit all the shared URLs as it could be pointing at places you don't want to request with your own IP. Let's check the module's options to see what we can do:</p> <div class="screenshot"> <a class="fancybox" href="/images/birdwatcher/url_crawl_options.png" title="Viewing options for the URL Crawler module"> <img class="thumbnail" src="/images/birdwatcher/url_crawl_options_thumbnail.png" alt="" /></a> <small>Viewing options for the URL Crawler module</small> </div> <p>This module has a bit more options than the <code>users/import</code> module. None of them are required but the <code>PROXY_ADDR</code> and <code>PROXY_PORT</code> are definitely a good idea to configure. The options will instruct the module to request all URLs through a HTTP proxy to hide the origin of the request for your own safety and OPSEC. I personally have <a href="https://www.torproject.org/">Tor</a> installed and its SOCKS proxy exposed as an HTTP proxy with Polipo. Check out this <a href="https://www.marcus-povey.co.uk/2016/03/24/using-tor-as-a-http-proxy/">blog post</a> if you want to know how it's done.</p> <div class="screenshot"> <a class="fancybox" href="/images/birdwatcher/url_crawl.png" title="Crawling URLs for more information"> <img class="thumbnail" src="/images/birdwatcher/url_crawl_thumbnail.png" alt="" /></a> <small>Crawling URLs for more information</small> </div> <p>We configure the module to use a proxy and run it. It will steadily crunch through the URLs but it might take a while to finish, depending on your connection speed, proxy, <code>THREADS</code> setting and the amount of URLs to crawl. The first time you run this module it can take quite a long time as it needs to process a lot of URLs.</p> <h3>Getting Klout information</h3> <p>The <a href="https://klout.com/s/developers/v2">Klout API</a> can give us a lot of valuable information on users such as their <a href="https://klout.com/corp/score">Klout score</a> which can be used to find users with the most reach and influence, the general topics they are Tweeting about, and an influence graph which can tell us who each user is influencing and who they are being influenced by.</p> <p>The first module we need to run is the <code>users/klout_id</code> module. This module simply retrieves each user's Klout ID which is needed for all the other Klout related modules:</p> <div class="screenshot"> <a class="fancybox" href="/images/birdwatcher/klout_id.png" title="Retrieving user's Klout ID"> <img class="thumbnail" src="/images/birdwatcher/klout_id_thumbnail.png" alt="" /></a> <small>Retrieving user's Klout ID</small> </div> <p>Next we run the <code>users/klout_topics</code> module which retrieves the general topics that each user is Tweeting about such as <em>Technology</em>, <em>Hacking</em>, <em>Marketing</em>, <em>Information Security</em>, etc. Each topic will be saved in a table and referenced through a join table to users, to make it easy to retrieve users who Tweet, or don't Tweet, about a specific topic:</p> <div class="screenshot"> <a class="fancybox" href="/images/birdwatcher/klout_topics.png" title="Retrieving user's Klout topics"> <img class="thumbnail" src="/images/birdwatcher/klout_topics_thumbnail.png" alt="" /></a> <small>Retrieving user's Klout topics</small> </div> <p>If we want to know about how influential each user is, we can use the <code>users/klout_score</code> module to retrieve their Klout score. The score is calculated by Klout and is explained <a href="https://klout.com/corp/score">here</a>, but the higher the score the more influential the user is:</p> <div class="screenshot"> <a class="fancybox" href="/images/birdwatcher/klout_score.png" title="Retrieving user's Klout scores"> <img class="thumbnail" src="/images/birdwatcher/klout_score_thumbnail.png" alt="" /></a> <small>Retrieving user's Klout scores</small> </div> <p>From the output we can see that mikko is the most influential followed by hdmoore and thegrugq. The klout score will of course also be saved to the database to make querying based on Klout scores possible.</p> <p>Lastly we will run the <code>users/klout_influence</code> module to retrieve information about who our users are being influenced by and who they are influencing:</p> <div class="screenshot"> <a class="fancybox" href="/images/birdwatcher/klout_influence.png" title="Retrieving user's Klout influence"> <img class="thumbnail" src="/images/birdwatcher/klout_influence_thumbnail.png" alt="" /></a> <small>Retrieving user's Klout influence</small> </div> <h3>Making a word cloud</h3> <p>A great way to get a quick sense of what the users are talking about is to use the <code>statuses/word_cloud</code> module. The module can generate a classic weighted word cloud based on Tweets from all users, or a smaller selection, within a window of time. The module has quite a lot of options for customization:</p> <div class="screenshot"> <a class="fancybox" href="/images/birdwatcher/word_cloud_options.png" title="Viewing options for the Word Cloud module"> <img class="thumbnail" src="/images/birdwatcher/word_cloud_options_thumbnail.png" alt="" /></a> <small>Viewing options for the Word Cloud module</small> </div> <p>We configure the module with a file destination for the generated image and set <code>INCLUDE_PAGE_TITLES</code> to true in order to mix in the page titles we previously retrived with the <code>urls/crawl</code> module. This gives an even better idea of the topics our users have been talking about over the last seven days:</p> <div class="screenshot"> <a class="fancybox" href="/images/birdwatcher/word_cloud.png" title="Generating a word cloud from Tweets"> <img class="thumbnail" src="/images/birdwatcher/word_cloud_thumbnail.png" alt="" /></a> <small>Generating a word cloud from Tweets</small> </div> <p>The result is a pretty word cloud that tells us what has been on our lovely horse's mind over the last seven days:</p> <div class="screenshot"> <a class="fancybox" href="/images/birdwatcher/word_cloud_result.png" title="The result of the word cloud module"> <img class="thumbnail" src="/images/birdwatcher/word_cloud_result_thumbnail.png" alt="" /></a> <small>The result of the word cloud module</small> </div> <h3>Generating an influence graph</h3> <p>The raw influence data we retrieved earlier with <code>users/klout_influence</code> can be visualized and examined with the <code>users/influence_graph</code> module:</p> <div class="screenshot"> <a class="fancybox" href="/images/birdwatcher/influence_graph.png" title="Generating a visual influence graph"> <img class="thumbnail" src="/images/birdwatcher/influence_graph_thumbnail.png" alt="" /></a> <small>Generating a visual influence graph</small> </div> <p>The result is a directional graph showing who is influencing who according to Klout:</p> <div class="screenshot"> <a class="fancybox" href="/images/birdwatcher/influence_graph_result.png" title="The influence graph visualized"> <img class="thumbnail" src="/images/birdwatcher/influence_graph_result_thumbnail.png" alt="" /></a> <small>The influence graph visualized</small> </div> <h3>Generating a social graph</h3> <p>Another type of graph we can generate is a social graph that doesn't use Klout's influence data but instead finds social connections by analyzing each user's Tweets for mentions of other users:</p> <div class="screenshot"> <a class="fancybox" href="/images/birdwatcher/social_graph.png" title="The social graph between users"> <img class="thumbnail" src="/images/birdwatcher/social_graph_thumbnail.png" alt="" /></a> <small>The social graph between users</small> </div> <p>The resulting graph is a bit different from the influence graph and shows a very tightly coupled cluster between some users. The edge weight between users is calculated simply by counting how many times they mention each other in Tweets. The thicker the line, the stronger the connection between two users:</p> <div class="screenshot"> <a class="fancybox" href="/images/birdwatcher/social_graph_result.png" title="The social graph visualized"> <img class="thumbnail" src="/images/birdwatcher/social_graph_result_thumbnail.png" alt="" /></a> <small>The social graph visualized</small> </div> <h3>Plotting a user's Twitter engagement</h3> <p>Another question you might ask is at what day and time a user is most engaged with Twitter. This might be useful for finding the time where a user is most likely to engage with you on Twitter. We can use the <code>users/activity_plot</code> module to get an idea of this:</p> <div class="screenshot"> <a class="fancybox" href="/images/birdwatcher/activity_plot.png" title="The social graph between users"> <img class="thumbnail" src="/images/birdwatcher/activity_plot_thumbnail.png" alt="" /></a> <small>Generating an activity plot for <em>halvarflake</em></small> </div> <p>The resulting plot tells us that <em>halvarflake</em> is generally very engaged with Twitter on Fridays at around 8AM and Tuesdays &amp; Wednesdays at around 7PM:</p> <div class="screenshot"> <a class="fancybox" href="/images/birdwatcher/activity_plot_result.png" title="Halvarflake's activity plot"> <img class="thumbnail" src="/images/birdwatcher/activity_plot_result_thumbnail.png" alt="" /></a> <small><em>halvarflake</em>'s activity plot</small> </div> <h3>Listing shared URLs</h3> <p>The last module I want to demonstrate in this blog post is the <code>urls/most_shared</code> module. The module will simply list URLs shared within a specific window of time ordered from most to least shared. If a URL has been shared by several users it is a pretty good indicator that it has interesting information:</p> <div class="screenshot"> <a class="fancybox" href="/images/birdwatcher/most_shared.png" title="Paging through the shared URLs"> <img class="thumbnail" src="/images/birdwatcher/most_shared_thumbnail.png" alt="" /></a> <small>Paging through the shared URLs</small> </div> <p>Because we ran the <code>urls/crawl</code> module earlier we also see page title, content type and HTTP code which is very convenient. Because I used Tor as an HTTP proxy we also ran into a CloudFlare CAPTCHA wall.</p> <h2>Wrapping up</h2> <p>This concludes my first post on Birdwatcher. I hope you enjoyed it and hope you will include it in your OSINT toolbox. Feel free to file any bugs on <a href="https://github.com/michenriksen/birdwatcher/issues/new">GitHub</a> or give me ideas for new modules.</p> <p>In the next blog post I will go over some of the more advanced functionality of Birdwatcher, like querying the underlying database for data, interacting with Birdwatcher's code through the interactive Ruby shell and how to write a new module.</p> tag:michenriksen.com,2016-03-12:/blog/new-version-of-gitrob-is-out/ A new version of Gitrob is out 2016-03-12T13:37:00Z 2016-03-12T13:37:00Z <%= render '/post_head.html', :post => item %> <p>It's been more than a year since I released the initial version of Gitrob. I haven't had a whole lot of time to expand on it, but now a new and improved version is finally here with a bunch of new features requested by users.</p> <p>For those who don't know, Gitrob is a command line tool which can help organizations and security professionals find sensitive information lingering in publicly available files on GitHub. The tool will iterate over all public organization and member repositories and match filenames against a range of patterns for files that typically contain sensitive or dangerous information. Head over to <a href="/blog/gitrob-putting-the-open-source-in-osint/">my previous post</a> for more details and screenshots, but be sure to come back here to learn about the new features!</p> <p>The attention the tool has received has been way over my expectations and I want to thank everyone who has helped spread the word about it. With over 1100 Stars on GitHub, a place in <a href="https://www.thoughtworks.com/radar/tools"> ThoughtWorks' Tech Radar</a> and a mention in <a href="http://www.amazon.com/The-Hacker-Playbook-Practical-Penetration/dp/1512214566">The Hacker Playbook 2</a>, I will strive to maintain and build upon the tool more frequently from now on.</p> <h2>New features</h2> <h3>Analyze arbitrary amount of organizations and users</h3> <p>The old version of Gitrob only allowed to analyze a single GitHub organization at a time, but the new version allows to mix any number of organizations and users in a single assessment. This is great if a company has multiple organizations or if you have identified GitHub users who work for the target company but don't have their membership publicly visible.</p> <div class="screenshot"> <a class="fancybox" href="/images/gitrob_1_0_0/analyze.png" title="Analyzing an organization and a user in one assessment"> <img class="thumbnail" src="/images/gitrob_1_0_0/analyze_thumbnail.png" alt="" /></a> <small>Analyzing an organization and a user in one assessment</small> </div> <h3>Create and delete assessments in web interface</h3> <p>The new version has an improved web application which allows users to run a new assessment directly from the web interface. This is very convenient if Gitrob runs on a server accessible to multiple users as they no longer require command line access to manage assessments.</p> <div class="screenshot"> <a class="fancybox" href="/images/gitrob_1_0_0/new_assessment.png" title="Creating a new assessments from the web interface"> <img class="thumbnail" src="/images/gitrob_1_0_0/new_assessment_thumbnail.png" alt="" /></a> <small>Creating a new assessments from the web interface</small> </div> <h3>GitHub Enterprise support</h3> <p>The new version makes it possible to run Gitrob against custom GitHub Enterprise installations by simply providing the location when creating a new assessment. <strong>Note:</strong> I unfortunately don't have access to a GitHub Enterprise installation, so I would appreciate if anyone could verify if this works and report any bugs!</p> <div class="screenshot"> <a class="fancybox" href="/images/gitrob_1_0_0/new_assessment_enterprise.png" title="Assessment against a custom GitHub Enterprise installatation"> <img class="thumbnail" src="/images/gitrob_1_0_0/new_assessment_enterprise_thumbnail.png" alt="" /></a> <small>Assessment against a custom GitHub Enterprise installation</small> </div> <h3>Compare assessments for continuous monitoring</h3> <p>In the old version it wasn't really easy to continuously monitor an organization, but the new version makes it possible to compare two assessments to quickly identify new or modified files, users and repositories.</p> <div class="screenshot"> <a class="fancybox" href="/images/gitrob_1_0_0/comparison.gif" title="Comparing two assessments"> <img class="thumbnail" src="/images/gitrob_1_0_0/comparison_thumbnail.png" alt="" /></a> <small>Comparing two assessments</small> </div> <h3>Highlighting of interesting values and detection of test files</h3> <p>In the small improvements category, the new version detects and highlights interesting values such as IP addresses, domains, tokens and email addresses when viewing a file's content. The new version will also attempt to determine if a file is likely test or mock related and make them less visible so they can easily be skipped.</p> <div class="screenshot"> <a class="fancybox" href="/images/gitrob_1_0_0/highlighting.gif" title="Highlighting of interesting values and dimming of test related files"> <img class="thumbnail" src="/images/gitrob_1_0_0/highlighting_thumbnail.png" alt="" /></a> <small>Highlighting of interesting values and dimming of test related files</small> </div> <h2>Under the hood</h2> <p>Apart from the new features, Gitrob v1.0.0 is pretty much a complete rewrite of the first version and some components have been switched out with better and more stable components. Gitrob now uses <a href="https://rubygems.org/gems/sequel">sequel</a> for database communication, <a href="https://rubygems.org/gems/github_api">github_api</a> for GitHub API interaction and <a href="https://rubygems.org/gems/thor">thor</a> for the command line interface. The switch to thor also means that the interface is a little different from the old version. <code>gitrob -o acme</code> is now <code>gitrob analyze acme</code>.</p> <h2>New file signatures</h2> <p>The new version also ships with a bunch of new signatures for files that might contain sensitive information:</p> <ul> <li>SSH configuration files <small>(Might contain usernames and SSH hostnames)</small></li> <li>PostgreSQL password files <small>(Contains PostgreSQL database credentials)</small></li> <li><a href="https://aws.amazon.com/cli/">AWS CLI</a> credential files <small>(Contains Amazon Web Services credentials)</small></li> <li><a href="http://dayoneapp.com/">Day One</a> journal files <small>(Might contain sensitive and personal information)</small></li> <li><a href="https://maebert.github.io/jrnl/">jrnl</a> journal files <small> (Might contain sensitive and personal information)</small></li> <li><a href="https://github.com/pearkes/tugboat">Tugboat</a> configuration files <small>(Might contain <a href="https://www.digitalocean.com/">DigitalOcean</a> credentials)</small></li> <li><a href="https://git-scm.com/docs/git-credential-store">git-credential-store</a> configuration files <small>(Contains Git credentials)</small></li> <li>Git configuration files <small>(Contains names, email addresses and occasionally access tokens)</small></li> <li><a href="https://www.chef.io/chef/">Chef</a> Knife configuration files <small>(Might contain references to Chef servers)</small></li> <li><a href="https://www.chef.io/chef/">Chef</a> private keys <small>(Gives access to Chef servers)</small></li> <li><a href="https://cpanel.com/">cPanel</a> backup ProFTPd credential files <small>(Might contain FTP server credentials)</small></li> <li><a href="https://robomongo.org/">Robomongo</a> configuration files <small>(Might contain MongoDB database credentials)</small></li> <li><a href="https://filezilla-project.org/">FileZilla</a> configuration files <small>(Might contain FTP server credentials)</small></li> <li><a href="https://filezilla-project.org/">FileZilla</a> recent servers files <small>(Might contain FTP server credentials)</small></li> <li><a href="http://www.ventrilo.com/">Ventrilo</a> server configuration files <small>(Might contain server credentials)</small></li> <li><a href="https://www.docker.com/">Docker</a> configuration files <small>(Might contain credentials for public or private Docker registries)</small></li> <li><a href="https://www.npmjs.com/">NPM</a> configuration files <small>(Might contain credentials for NPM registries)</small></li> </ul> <p>Check out <a href="https://github.com/michenriksen/gitrob/blob/master/signatures.json">signatures.json</a> for the full list of file signatures.</p> <p>If you have a good idea for a new signature, please don't hesitate to make a Pull Request or simply create an Issue with details and I will look into it!</p> <p>I am very excited about this release and I hope you are too. Hurry up and run Gitrob against your organization before someone else does! Installation and setup instructions can be found in the <a href="https://github.com/michenriksen/gitrob">README on GitHub</a>.</p> <p><strong>have fun and be responsible!</strong></p> tag:michenriksen.com,2015-01-12:/blog/gitrob-putting-the-open-source-in-osint/ Gitrob: Putting the Open Source in OSINT 2015-01-12T13:37:00Z 2015-01-12T13:37:00Z <%= render '/post_head.html', :post => item %> <div class="alert alert-info" role="alert"> <strong>Heads up!</strong> A new version of Gitrob is out with a bunch of improvements and features! <a href="/blog/new-version-of-gitrob-is-out/">Read more here</a>. </div> <p>Developers generally like to share their code, and many of them do so by open sourcing it on <a href="https://github.com">GitHub</a>, a social code hosting and collaboration service. Many companies also use GitHub as a convenient place to host both private and public code repositories by creating GitHub organizations where employees can be joined.</p> <p>Sometimes employees might publish things that should not be publicly available. Things that contain sensitive information or things that could even lead to direct compromise of a system. This can happen by accident or because the employee does not know the sensitivity of the information.</p> <p><a href="https://github.com/michenriksen/gitrob">Gitrob</a> is a command line tool that can help organizations and security professionals find such sensitive information. The tool will iterate over all public organization and member repositories and match filenames against a range of patterns for files that typically contain sensitive or dangerous information.</p> <h3>How it works</h3> <p>Looking for sensitive information in GitHub repositories is not a new thing, it has been <a href="http://blog.conviso.com.br/2013/06/github-hacking-for-fun-and-sensitive.html" rel="nofollow">known for a while</a> that things such as private keys and credentials can be found with GitHub's search functionality, however Gitrob makes it easier to focus the effort on a specific organization.</p> <p>The first thing the tool does is to collect all public repositories of the organization itself. It then goes on to collect all the organization members and their public repositories, in order to compile a list of repositories that might be related or have relevance to the organization.</p> <a class="fancybox" href="/images/gitrob/gitrob_collecting_repos.png" title="Gitrob collecting repositories from organization members."> <img class="thumbnail" src="/images/gitrob/gitrob_collecting_repos_thumb.png" alt="" /> </a> <p>When the list of repositories has been compiled, it proceeds to gather all the filenames in each repository and runs them through a series of observers that will flag the files, if they match any patterns of known sensitive files. This step might take a while if the organization is big or if the members have a lot of public repositories.</p> <a class="fancybox" href="/images/gitrob/gitrob_analyzing_repos.png" title="Gitrob sifting through collected repositories and flagging interesting files."> <img class="thumbnail" src="/images/gitrob/gitrob_analyzing_repos_thumb.png" alt="" /> </a> <p>All of the members, repositories and files will be saved to a PostgreSQL database. When everything has been sifted through, it will start a <a href="http://www.sinatrarb.com">Sinatra</a> web server locally on the machine, which will serve a simple web application to present the collected data for analysis.</p> <a class="fancybox" href="/images/gitrob/gitrob_findings.png" title="Interesting files across all repositories are shown in one list for easy analysis. The quick filter in the top right corner can be used to look for specific files."> <img class="thumbnail" src="/images/gitrob/gitrob_findings_thumb.png" alt="" /> </a> <a class="fancybox" href="/images/gitrob/gitrob_findings_details.png" title="Clicking on a file will show its contents with syntax highlighting. It will also show why the file was flagged."> <img class="thumbnail" src="/images/gitrob/gitrob_findings_details_thumb.png" alt="" /> </a> <a class="fancybox" href="/images/gitrob/gitrob_members.png" title="Members of the organization can be viewed in a grid layout. Members with interesting files are easy to spot."> <img class="thumbnail" src="/images/gitrob/gitrob_members_thumb.png" alt="" /> </a> <a class="fancybox" href="/images/gitrob/gitrob_members_details.png" title="Clicking on a member will show their basic information and public repositories. Repositories with findings are highlighted with an orange background."> <img class="thumbnail" src="/images/gitrob/gitrob_members_details_thumb.png" alt="" /> </a> <a class="fancybox" href="/images/gitrob/gitrob_repos.png" title="All collected repositories can be viewed in a table with their descriptions and website URLs. Repositories with findings are highlighted with an orange background."> <img class="thumbnail" src="/images/gitrob/gitrob_repos_thumb.png" alt="" /> </a> <a class="fancybox" href="/images/gitrob/gitrob_repo.png" title="All files in a specific repository can be viewed. The quick filter in the top right corner can be used to look for specific files."> <img class="thumbnail" src="/images/gitrob/gitrob_repo_thumb.png" alt="" /> </a> <h3>Some findings</h3> <p>While developing Gitrob, I tested it against many different organizations belonging to various companies, big and small, both to expose the tool to a lot of real-life data and to notify the companies of any findings before release.</p> <p>The tool found several interesting things ranging from low-level, to bad and all the way to company-destroying kind of information disclosure. Here's some examples...</p> <p><strong>Note:</strong> I have redacted sensitive and identifying information in the screenshots; I am not interested in embarrassing or exposing anyone. And again, all these findings have been reported.</p> <a class="fancybox" href="/images/gitrob/finding1.png" title="Found in a <code>.bash_profile</code> file, the employee was thoughtful enough to mask the passwords, but still mapped out a big chunk of infrastructure with his command aliases. It also tells attackers that spear-phishing this employee will likely give them root access to a lot of databases."> <img class="thumbnail" src="/images/gitrob/finding1_thumb.png" alt="" /> </a> <a class="fancybox" href="/images/gitrob/finding2.png" title="Found in a <code>.bash_profile</code> file, the command aliases revealed the existence of a secret <em>black site</em> domain used for the company's tools for everyday operations such as analytics, metrics and continuous integration. A big increase in attack surface."> <img class="thumbnail" src="/images/gitrob/finding2_thumb.png" alt="" /> </a> <a class="fancybox" href="/images/gitrob/finding3.png" title="Command history files can contain a lot of sensitive information, such as passwords, API keys and hostnames."> <img class="thumbnail" src="/images/gitrob/finding3_thumb.png" alt="" /> </a> <a class="fancybox" href="/images/gitrob/finding4.png" title="A developer had open sourced a Wordpress website, including a complete database dump with password hash for his user account. Maybe the same password is used somewhere else?"> <img class="thumbnail" src="/images/gitrob/finding4_thumb.png" alt="" /> </a> <a class="fancybox" href="/images/gitrob/finding5.png" title="An <code>.env</code> file for a chat bot contained several credentials. Apart from an attacker being able to spy on their Campfire chat and steal stuff from the data stores, they would also be able to control the temperature somewhere with the Nest credentials."> <img class="thumbnail" src="/images/gitrob/finding5_thumb.png" alt="" /> </a> <a class="fancybox" href="/images/gitrob/finding6.png" title="A company had open sourced their documentation website, a simple Ruby On Rails application. They forgot to remove the application secret token, which can be <a href='http://www.rapid7.com/db/modules/exploit/multi/http/rails_secret_deserialization'>exploited</a> to achieve remote code execution."> <img class="thumbnail" src="/images/gitrob/finding6_thumb.png" alt="" /> </a> <a class="fancybox" href="/images/gitrob/finding7.png" title="A developer had checked in his KeePass password database containing 174 entries. The data is heavily encrypted, but the master password can be <a href='https://github.com/darkk/keebrute'>brute-forced</a>. In this case the company was certainly interesting enough for someone to throw a lot of computing power at that task."> <img class="thumbnail" src="/images/gitrob/finding7_thumb.png" alt="" /> </a> <a class="fancybox" href="/images/gitrob/finding8.png" title="Amazon EC2 credentials found in a <code>.zshrc</code> file. depending on the level of privilege, it can potentially give complete control of the company's infrastructure."> <img class="thumbnail" src="/images/gitrob/finding8_thumb.png" alt="" /> </a> <a class="fancybox" href="/images/gitrob/finding9.png" title="An employee had checked in an Amazon EC2 private key which can potentially give complete control of the company's infrastructure."> <img class="thumbnail" src="/images/gitrob/finding9_thumb.png" alt="" /> </a> <a class="fancybox" href="/images/gitrob/finding10.png" title="The same employee from the last screenshot also checked in his private SSH key, which could potentially grant access to the company's SSH servers. It could potentially also be used to clone private organization repositories."> <img class="thumbnail" src="/images/gitrob/finding10_thumb.png" alt="" /> </a> <h3>Installing and setting up Gitrob</h3> <p>Gitrob is written in Ruby and requires at least version 1.9.3 or above. If you are on an older version, it is very easy to install newer versions with <a href="http://rvm.io/">RVM</a>. If you are installing Gitrob on <a href="http://www.kali.org/">Kali</a>, you are almost good to go, you just need to update Bundler with <code>gem install bundler</code>. It might also be necessary to install a PostgreSQL dependency with <code>apt-get install postgresql-server-dev-9.1</code> in a terminal.</p> <p>Gitrob is a Ruby gem, so installation is a simple <code>gem install gitrob</code> in a terminal. This will automatically install all the code dependencies as well.<p> <p>A <a href="http://www.postgresql.org/">PostgreSQL</a> database is also needed for Gitrob to store its data. Installing PostgreSQL is pretty straight forward; here is an installation guide for <a href="http://www.gotealeaf.com/blog/how-to-install-postgresql-on-a-mac">Mac OS X</a> and one for <a href="https://www.digitalocean.com/community/tutorials/how-to-install-and-use-postgresql-on-ubuntu-14-04">Ubuntu/Debian</a> based Linux. If you're installing Gitrob on Kali, you already have PostgreSQL installed, however you need to start the server with <code>service postgresql start</code> in a terminal.</p> <p>When PostgreSQL is installed, it's time to create a user and a database for Gitrob. To do so, type the following commands in a terminal:</p> <pre> sudo su postgres # Not necessary on Mac OS X createuser -s gitrob --pwprompt createdb -O gitrob gitrob </pre> <p>The last thing we need is a GitHub access token in order to be able to talk to their API. The easiest way is to create a <a href="https://github.com/settings/applications">personal access token</a>. If you plan on using Gitrob extensively or on a very big organization, it might be necessary turn down the amount of threads used and maybe configure Gitrob to use access tokens from you and your colleagues, to avoid running into rate limiting.<p> <p>When everything is ready, simply run <code>gitrob --configure</code> and you will be presented with a configuration wizard that asks you for database connection details and GitHub access tokens. All of this configuration can be changed by running the same command again. The configuration will be saved in <code>~/.gitrobrc</code> - and yes, Gitrob is looking for this file too so watch out.</p> <a class="fancybox" href="/images/gitrob/gitrob_configuration.png" title="Setting up Gitrob with the configuration wizard."> <img class="thumbnail" src="/images/gitrob/gitrob_configuration_thumb.png" alt="" /> </a> <p>When everything is set up, you can start analyzing organizations by running <code>gitrob -o &lt;orgname&gt;</code> in a terminal. To see options, use <code>gitrob --help</code>.</p> <h3>Why I created Gitrob</h3> <p>I work in the security team at <a href="https://soundcloud.com/">SoundCloud</a> (<a href="https://soundcloud.com/jobs">We're hiring</a>, btw) and one of my recent tasks has been to create a system that continuously watches our GitHub organization for various things that might be a security risk, including looking for potential sensitive files in repositories. During development, I thought it would be interesting to take parts of this system and open sourcing it as a tool that can be used both defensively and offensively.</p> <p>If you are responsible for security at a company that uses GitHub for hosting code, Gitrob can be used to periodically check your organization for any sensitive files that might be lingering in repositories.</p> <p>If you are on the offensive side, like a professional penetration tester, Gitrob can be used in the initial information gathering stage to look for anything that might give you a foothold or increase the target's attack surface. Gitrob can also give you usernames, names, email addresses and names of internal systems that are useful in phishing campaigns and social engineering attacks. If you are lucky, Gitrob can even give you complete pwnage without ever sending a single malicious packet to the target's systems.</p> <h3>Helping out</h3> <p>Gitrob should be considered Beta and there is probably a good amount of bugs. <a href="https://github.com/michenriksen/gitrob/issues">Bug reports</a> and suggestions for improvements are welcome!</p> <p>Another way to help out is to contribute new patterns for sensitive files. If you know of any sensitive files that are not already identified, please submit them in a pull request on <a href="https://github.com/michenriksen/gitrob">GitHub</a>. I am especially interested in sensitive web framework files and configuration files. Have a look at the <a href="https://github.com/michenriksen/gitrob/blob/master/signatures.json">signatures.json</a> file to see what is already looked for.</p> <p><strong>Have fun and be responsible!</strong></p>