Our keynote to start the day was delivered by Raphael Perl, Head of the Action Against Terrorism Unit, Organization for Security and Co-operation in Europe (what’s with these titles? sheesh) who discussed global cybersecurity challenges for practitioners as well as the emerging threats and tactics of terrorism in cyberspace.

One of the items I thought most interesting was the explanation of overall approach on a political and military level. From a political perspective, it’s becoming understood and accepted that security is a global challenge requiring international information sharing. But the military is resisting, instead preferring to stay separate and secretive. As I pointed out on day 1, Philip Reitinger and others plainly state the biggest issue we face is hackers and other baddies becoming much more organized and globally sharing information at a rate the good guys can’t match. It would seem to me the military approach is destined for failure, and Raphael Perl made it clear he believes the same. Here’s a few other interesting points or facts from the discussion:

• Last June, 1 billion PC’s were in use worldwide. By 2012 this will double to 2 billion.
• 200 billion email messages are sent every day, Cisco estimates 90% of them are SPAM.
• There are currently 3-5000 active websites run by terrorists or are terrorist affiliated.
• In a non-scientific poll of sales at a major computer store, 10% of home users bought antivirus software with a new PC purchase. Business users bought it 90% of the time.

He devoted a lot of time to discussing the potential threat to cyberspace by terrorists. He believes terrorists are actively planning to disrupt the Internet, and will launch an attack in conjunction with a real attack or a major national disaster. The goal is interrupting services right when many need it most. He believes the information sharing and readily-available tools by hackers will help facilitate these attacks, as any cyberattack can be easily copied by terrorists. However, it was also made clear that not all experts agree with his opinion. Overall I found the discussion very intriguing and certainly worthy of deep future interest.

Day 2 tracks were Five Common Mistakes in Securing Web Applications, Are you Googling Your Privacy Away?, Are You Prepared for Data Loss, and PII (personally identifiable information): Taming the Beast (AKA how to discover and secure PII). Just like yesterday, they were all engaging and very well presented. The Google talk was particularly thrilling – yea I said thrilling. It’s incredible how much private information we give away to Google in exchange for their services. If you know me, you know I’m not afraid of talking about how much Google scares me. When it’s all laid out for you in a short presentation it’s even more resonating.

• Google is better than everyone else at screwing you and your privacy.
• Their cookies auto-renew every time you use any service.
• They’ve essentially built the best surveillance system ever created. You are a GUID (Google User ID), everything you do with them maps to this GUID, and it will follow you until you die. They aim to have a GUID for every person in the world.
• You readily give Google (and other social networking sites) vastly more information than the police can ask you.
• GMAIL really is email for life. According to the ECPA your data is only protected for 180 days, on day 181 Google can give it to law enforcement without a warrant, barely with a formal request. Our DOJ is working on depreciating the 180 day rule.
• Google has only indexed 20% of the internet.
• If you use Google Desktop with remote access, everything you have indexed is stored in the Googlesphere. You’re violating HIPAA, PCI, SOX, and most other compliance specifications (oops!).
• When using Chrome, Google knows every keystroke you type into the browser. Even fields you backspace/delete.
• They don’t have to ever delete anything you tell/ask them to. Google’s file system is designed not to delete.
• Does your phone run Andriod? They know everyone you talk to, where you are, where you’ve been.
• Goog411 is a slick, free 411 service, right? Actually, you’re training their voice recognition software.
• Google probably knows much more about you than any government agency.
• Remember that HIPAA, PCI, and SOX violation stuff? Hope you’re not using GMAIL since it’s stored (and indexed) in the Googlesphere.
• Google probably knows much more about you than any government agency.
• Yes, they can filter via your GUID and/or IP address and find all of your search terms.

I could go on, and the presentation was less than an hour. It’s freaking scary. Here’s the rest of my juicy tidbits from the day:

• 60% of top 100 websites had hosted or or were involved in malicious activity in 2008.
• Of all 2008 vulnerabilities, 58% were from web applications.
• From the 2008 total, 73% of those classed “easily exploitable” were web applications.
• Dear developers: Most hackers don’t use browsers to exploit web applications, client-side anything is fail.
• 80% of internet traffic crosses a Verizon network.
• In the vast majority of cases when a breach occurred due to a known software vulnerability, the patch was released for over a year.
• Top three types of malware are keyloggers, back doors, and capture-and-store programs.
• The majority of these are plain, un-customized and easily detectable by most antivirus programs.
• 49% of breaches go undiscovered for months.
• VerizonBusiness’ assessment of breach events over three years revealed 82% of organizations captured the attack(s) in logs, but the logs were either too complex or they lacked the tools to filter the data into a useful view.
• In this period, 69% of breaches were detected by a third party. 24% were detected internally passively (stumbling on), and 7% were detected actively.
• Over 260 million SSN’s have been leaked since 2005.
• There are a few open source tools for scanning/identifying PII.
• According to solution methodology, 89% of breaches could have been solved by data-at-rest protection (identification and removal or encryption, etc.)

I have 21 pages of notes and a large list of ideas/tasks taken over those two days. So let me say again what a fantastic event this conference was.

Chris

The morning began with welcoming remarks and an animated hacking demonstration themed around X-Men, which consisted of using a Linux distro to disable an NT password, Internet-mining to find information about an individual, and WireShark to sniff packets. Wasn’t exactly technically deep, but certainly not boring powerpoint stuff and if anything was pretty entertaining (Indian guy in a Wolverine wig, nice).

The keynote was delivered by Philip Reitinger, National Protection and Programs Directorate for the U.S. Department of Homeland Security (what a mouthful). Without powerpoint (bonus!), he discussed quite a bit on the 60-day Review(pdf) and where DHS is challenged with cybersecurity. He also candidly highlighted some of the largest challenges facing security professionals:

• Hackers getting better not just at hacking but with sharing information at a rate security professionals cannot match.
• Defenses not keeping pace with threats.
• Cybersecurity as an issue of national security (which the president recently accepted responsibility for, a great first step).
• Fostering public-private information sharing.
• Establishing reasonable metrics.

Overall a thoroughly interesting and engaging discussion. He also pushed pretty hard for good IT people, (my boss stop reading here) if you’re interested in moving into the Federal space, now might be a great time.

The three tracks I chose were Ensuring Network Protection While Meeting Compliance (PII, HIPAA, etc.), Computer Network Simulators, and Motivating People to Adopt Security Practices. You’d think these would be horribly boring, but I have to say how impressed I am not only with the conference organization but with the quality of the talks. Yes, really, I stayed engaged all day. Let me share a few tidbits of info I found particularly interesting throughout the day:

• Information Security Officers must be allowed a seat at the executive table and involved in business decisions. (there is a heavy push to remove the “wall” of security as a sub-position of IT or an afterthought)
• 35.7 million records potentially breached in 2008 *reported* – imagine what the actual number is. Dizzying.
• In 2008, missing or stolen equipment accounted for 42% of reported breach events – the second highest was employee negligence at 16%.
• Heathrow airport in England averages 900 unclaimed laptops per week – and after reasonable time unclaimed are auctioned off.
• 1 in 10 people click through SPAM and become infected with malware. On the surface, that’s not much. But think of an organization with 1000 or more people.
• There are 500,000 different variants of malware currently, 20,000 new ones are created every day.
• Personal observation: Most admins don’t have a clue how base32 encoded data looks (scary).
• People will not embrace security policies if they reduce their productivity, feel threatened, or are negatively reinforced. [Innovaites, take special note here, I listened well to this one ]

Overall I’ve found the conference very well organized, technically awesome, and the people very welcoming. I also had a few great conversations with some of the sponsoring vendors. So far a great experience, and I’m looking forward to tomorrow. Until then, cheers from the Albany Pump House and my beer sampler.

P.S. I’ve been tweeting some of the conference, feel free to follow me.

Chris

1) A prioritized monitoring plan.

Critical systems: A website, server, or critical VPN tunnel is hard down. Anything that should be up 24/7 is what matters. You need to know about this stuff immediately. You should be checking it every minute.

Non-critical systems: These can be addressed during business hours. A server running low on disk space or CPU at 100% because a backup is running? Your piecemeal desktop running eight virtual machines is down? Yes, they’re important. But while they look cool when you’re hangin’ at a LAN party and your phone is going nuts (I can hear you now “Nah man, it’s cool, just my monitoring server…”) or on some management report, you don’t need disturbed at 4AM for them.

Why separate your notifications? Your biggest enemy to emergency preparedness is you. If you constantly get text messages for everything possible, it’s no wonder you sleep through them. By knowing any text message you get may be a critical problem, you’ll be less likely to ignore it.

2) Great, unbiased monitoring.

While there are lots of internal monitoring packages, we like NAGIOS. It’s a bit of a bear to set up if you’re not a Linux’ite, but it provides nearly unlimited monitoring capability on many levels across almost all operating systems. Simple PING tests and HTTP checks to custom integrations, NAGIOS can do it with a little patience and best of all it’s completely free.

You should be monitoring externally too. This is especially true if you host websites or other external services. I spent a few months testing lots of external monitoring and uptime companies and chose Pingdom. They proved to be the most reliable, use many monitoring sites across the world, and had zero false alarms during the testing period – an experience competition four times their price cannot claim. They also provide a cool API for integrating uptime stats into your webpage and I can’t wait to turn the developers loose with that in the future.

Using both external and internal monitoring will provide a comfortable level of redundancy. Even if you just use Pingdom to check and make sure your NAGIOS server is up, have an external monitor! If you’re only using NAGIOS and the internet connection it uses to send you notifications (and host all of your websites) goes down, you’ve effectively just shot yourself. Use the 12-gauge.

Further words about external monitoring: You may be tempted to put your own NAGIOS box at your house on your cable or DSL connection for your external monitoring. Don’t. Sign up for Pingdom or another multiple-site unbiased monitoring service. We found that even with a business-class DSL connection and static IP address, notifications could be spotty due to the lackadaisical support of email by most ISP’s. Many short, identical looking emails with lots of IP addresses and time stamps in them look an awful lot like SPAM. Not to mention internet connections at most homes aren’t that reliable, which harkens back to #1 up there. If you’re constantly getting false notifications that’s the same as getting meaningless ones.

3) Make your notifications annoying!

You’ve got a plan and your notifications come from both the inside and outside, great! If you’ve got a cell phone specifically for work that you never text on, getting simple text messages to this device and setting a nice, annoying and LOUD sound for them may be enough. If you’re like many engineers out there (including me) who blend their professional and personal lives and every text message may not be an emergency (or you can sleep through a text message notification), you may need a little more than that.

The single biggest improvement I have made to Innova Partners network engineering operations is turning an emergency notification into a phone call.

That, my friend, is a bold statement. Enter eNotifyMe. This service is a gem, a true diamond-in-the-rough of the internet. Their single biggest feature is turning any email into a phone call. So now not only do your emergency notifications come as a text message, they come as a phone call. There has not been a better way to add urgency than this. It’s easy to not hear a few text messages – it is not easy to miss a few phone calls. Besides this great feature, eNotifyMe provides a ridiculous amount of triggers including AND/ALL/OR situations and schedules for notifications to phone, text, SMS, and more.

Excellent example: One of our clients has a phone server that can send all of its built-in problem notifications to only one email address. This is an issue for two reasons, all of the notifications it sends aren’t critical. In fact, 98% of what it sends are decidedly non-critical. The other is because it can send to only one address, I would have to set up a distribution group on our end to get the notifications where they need to go and I still cannot filter critical from non-critical. Using eNotifyMe I can filter the notifications and send them to the appropriate notification addresses (email for non-critical, phone/text/pager for critical). How cool is that!

Did I just say pager? Yep. Because our systems being up is so critical, I choose to have a redundant notification device for our engineers as well. Any text message that is sent to a cell phone is also sent to our pager. If we have a cell phone malfunction, dead battery, run over by a truck, we’ll still know if our systems are down. A pager can be cheap insurance for your department (about $40 a quarter per-pager) and while they’re not fool-proof because service can be spotty outside of metro areas, they can be a lifeline in case of emergency.

Finally, and probably most importantly, add your client facing staff to critical notifications. They don’t need phone calls, but make sure they’re getting emails and text messages. Chances are, if there is an emergency, your clients are going to call who they know to get the skinny – not you. They’ll pick up the phone and dial the cell of their sales representative way before they dial some 800 number into a support department. If your sales staff knows there is a problem and can soothe a client immediately, it won’t exacerbate the situation with everyone trying to get a hold of you while you have 20 notifications going off in your face. Not to mention your excellent preparedness and unified response is probably well ahead of your competition.

In a perfect world, you’ll never need any of this and it’ll all be a questionable line-item on a budget spreadsheet. The day you do (and you will) you’ll be glad you did. Even at 4AM on a Saturday.

They don’t get their parts from the same suppliers as most PC makers. No, they’re far choosier about the components that go into their machines, which is why they’re twice as much as any comparable PC. Which is why I will hold them to a higher standard, since the vast majority of their user-base is happy to smug you into suffocation about what pinnacles of quality they are. So forgive me if I get upset that the brand new THREE THOUSAND DOLLAR MacBook Pro which I bought for Kent has been a flaming turd since the day we opened the box. It grey screens (crashes) randomly, and has since minute one. Since day two the sound stared wigging out, making sounds like a ghost was getting raped with a freshly cut pine bough. Then after another day the sound just quit working (with much relief to the ghost, I’m sure). But you know, I know that even THREE THOUSAND DOLLAR computers can have a bad apple (heh) in the bunch. That’s why we bought the advanced AppleCare plan, right?

Wrong. AppleCare is the worst support experience I’ve ever had. I paid $349 to call a DMV owned by Steve Jobs. Seriously. Let’s start with their support hours – 6a-6p PST (that’s 9a-9p for those of us in EST). Wait, what? They have support hours to begin with? Why they don’t have 24/7 support simply escapes me. How can they possibly even try to be a contender in the global PC market, specifically to businesses, with support hours? I know that if I want to call Dell at 3am on Christmas morning while sitting in my underwear at home just because I want to talk to someone, they’ll be there. You know what’s better? I can call them, and with their next business day service (that costs less than AppleCare for a three year extension) I’ll have parts and/or a technician on my doorstep on December 26th. That’s the kind of service it takes to be a global player. But let’s just hope their computers don’t run the call-center…

Because I got hung up on. Yep, the first time I called in (right at 9am with a two minute wait), I started to talk with the support person and then got bounced to hold music followed by a hang-up one minute later. Nice. So I called back and now my wait was TWENTY MINUTES. Twenty minutes. For a THREE THOUSAND DOLLAR computer. I had to wait twenty minutes (to be fair, they picked up in nineteen minutes forty-seven seconds) to speak with someone after being hung up on. I’ve never had to wait to speak to someone at Dell. Ever. Once you get through the menus you talk to someone. How can Apple, the crusaders of “friendly” computing, not have enough staff to serve you within three minutes?

They’re too busy insulting their customers. That’s right. Not only did the gentleman I spoke with sound like he’d been toking-up right before his shift, he was downright rude and full of attitude. I actually had to tell him to cut out the attitude. I don’t like having to tell the person I’m calling for support to reduce their attitude.

Not that it did me any good; I still have to wait to get the notebook repaired. Our AppleCare Protection Plan doesn’t have a next day service, nor does it have a replacement service for a brand new THREE THOUSAND DOLLAR computer. Dell does, for less than the price of AppleCare. Your options are mailing it in and get it back “Uh, maybe a week or so?” (quote from my support representative) later or you can take it directly to the closest Apple Store which “Uh, might be faster than mailing it in”. So now not only have I lost the productivity of my web developer, I get to lose a nice chunk of my time as well shuffling the THREE THOUSAND DOLLAR computer to the other side of town. And I get to wait for my support appointment this afternoon, because you can’t even take it in at your leisure. God forbid you actually need to get YOUR computer fixed on YOUR schedule.

Apparently Apple truly does “think different”. Unless they start thinking more like Dell’s support department then I pity any professional who has an issue with their PC. God forbid trying to administer an office full of these things. Let’s hope my experience at the store today is better, 1:15pm sharp. I think they flog you with a wet cane if you miss the appointment.

Now, let me tell you what I’m not going to address with this post. Pretty much everything about Trixbox. Matt hassled me for a blog entry, and since I’m still testing and experiencing Vista, he said something to the tune of “You know, the phone system would make a great blog post…” So you’re going to get one tidbit about Trixbox – though it’s the most important tidbit you need if you’re going to use Bandwidth.com for your IP trunks. Unfortunately, doing a post on how to set up Trixbox is significantly beyond the scope of a simple blog. As a matter of fact the best guides I found are over 100 pages. However, I spent a huge chunk of time trying to figure out the correct settings for use with Bandwidth.com because their Asterisk/Trixbox support sucks, and no one else on the internet really seemed to get it either. Hopefully I can save you some time with how I got my trunks to work the way I want.

Setting up your SIP trunk(s):

General Settings

Outbound Caller ID: “Entity Name” <1xxxxxxxxxx>

This should be your company name and main number. Don’t worry, with these settings you’ll be able to override this global caller ID setting if you want.

Never Override CallerID: unchecked

Maximum channels: x

This is the number of channels you have purchased from Bandwidth.com

Disable Trunk: unchecked

Monitor Trunk Failures: not enabled

Outgoing Dial Rules

Dial Rules: 1+NXXNXXXXXX 1614+NXXXXXX

Bandwidth.com (and almost every other provider) requires full 11-digit numbers for outgoing calls; these are my rules to change user-dialed local and long-distance/toll free numbers without the 1 or 1+area code into 11-digit numbers. If you want to force your users to dial all 11 digits, don’t put these rules in and they’ll always fail unless they dial the whole number.

Outbound Dial Prefix: +

This setting is EXTREMELY IMPORTANT, Bandwidth.com requires a + in front of outgoing numbers sent to them. Without this your outgoing calls will fail.

Outgoing Settings

Trunk Name:

Whatever you want to name your trunk, something short and descriptive like Bandwidth-1 would be sufficient.

PEER Details: allow=ulaw dtmfmode=rfc2833 fromdomain=xxx.xxx.xxx.xxx host=xxx.xxx.xxx.xxx nat=yes port=5060 type=peer

fromdomain and host are both the IP that Bandwidth.com gives you for SIP traffic. I also set nat=yes because I’m NAT’ing through our super-badass firewalls. If you’re not doing NAT, I think you can set this to no or just don’t include it.

Incoming Settings

USER Context:

Whatever you want, this setting will be over-ridden in the details

USER Details: canreinvite=yes context=from-pstn dtmfmode=rfc2833 fromdomain=xxx.xxx.xxx.xxx host=xxx.xxx.xxx.xxx insecure=very nat=yes port=5060 type=peer

Registration

Register String:

Bandwidth.com does not need a register string, so this setting is blank.

Overriding caller ID per extension:

This is hella-easy. When you create the extension in FreePBX, enter the Outbound CID like this:

“Entity Name” <1xxxxxxxxxx>

Whatever you enter here will be displayed on the outgoing call instead of the global value set in the trunk. If you leave it blank, the global value will be used.

A few words about NAT…

Your Trixbox is behind a firewall, and you’re only getting one-way sound, huh? You need to create the following NAT rules in your firewall:

SIP: Port 5060 TCP and UDP from the IP’s Bandwidth.com gave you to the internal IP of your Trixbox.

RTP: Ports 1024-64000 UDP from any IP to the internal IP of your Trixbox.

SIP is the carrier of your call signal, and RTP is the media (voice) connection. Bandwidth.com uses several providers for RTP, you have to leave it wide open since it will be coming from always-changing IP’s. But as long as you have your rule for SIP locked down to the IP’s that Bandwidth.com gave you everything should be fine. Your Trixbox will not accept media without the SIP carrier.

Now you need to make two entries into your sip_nat.conf file:

externip = xxx.xxx.xxx.xxx

localnet=xxx.xxx.xxx.xxx/xxx.xxx.xxx.xxx

externip is your external IP address and localnet is your internal network/subnet mask. You can use two or more localnet entries depending on how many local network segments you use.

I think that’s mostly the gist of it. If you run across this post and have a question, feel free to comment and I’ll do my best to help. If I can think of any other big pains I ran across, I’ll blog them too.

P.S. I learned how to use the more tag in this post. There is hope for me!