Archive for the ‘Articles’ Category

Odd bugs and the case of an aging database

Sunday, August 29th, 2010

I was doing some troubleshooting recently trying to fix some issues reported in the Captcha of a registration system. The Captcha wasn’t always properly displaying and wouldn’t recognize valid code entries intermittently. After tracing some of the PHP to find just how the Captcha’s were being generated and managed, I found my way to the MySQL database and the table storing the values.

It took only a glance at the last and only row  to determine what the problem was – The primary key for the Captcha table was defined as an INT and had reached the value 65535. With the key unable to increment, any non-valid attempt (be it a bot or someone who couldn’t read the Captcha properly), would cause the system to lock up until the 1 in 100,000 chance that the same value was re-generated and someone correctly matched the code.  The failure resulting from this oversight was essentially a Denial of Service on the registration system.  To temporarily solve the issue, I did a simple reset of the primary key[1], which should buy a few years of development time to resolve the core issue.

This situation possesses an interesting design consideration when building long-running applications, even when you aren’t planning on storing a large amount of data for a long time. Tables that are frequently populated with new data should have measures in place to either handle recycling over-used values or should be built in a way that avoids these sort of conditions.  In the case of the Captcha above, the primary key was hardly used and could have been factored out entirely without the application running differently.

Looks like it’s time to get this database a subscription to AARP and some Depends….

References:

[1] Resetting a Primary Key

Tags: , , , , , , ,
Posted in Articles | No Comments »

Thoughts on IPv6 Firewalling

Monday, August 9th, 2010

It is the on-coming storm. We’ve all heard about it; it’s big, scary, and new. Public IP addresses for all and tearing down those NAT-PT barriers excites and worries many. IPv6 is coming but it’s not as frightening as it’s made out to be; most concepts from IPv4 can be applied directly to it’s younger brother. One of the ideals that IPv6 was designed around is the restoration of end-to-end connectivity of all nodes on the Internet; this introduces a certain uncertainty to how we design our firewalls under v6. We have options now, from the completely open networks geeks would love, to NATesque heavily stateful firewalls, and everything in between.

With our current version of the Internet Protocol we use private addresses heavily on internal networks protected automatically by the one way nature of NAT-PT; it has given many of us the expectation that connections cannot venture into our networks without our explicit permission. These networks are often described as castles or walled gardens, analogies referring to the strong perimeters and minmally secured interiors. We feel comfortable and secure in our walled gardens and IPv6 seems like it may rip some of that security away but this need not be the case. Regardless, the IPv4 internet is running out of addresses and IPv6 is coming; as the boyscout motto goes, ‘Be prepared’.

The paradigm of NAT is no longer the default in IPv6, the default has become every node exposed to the entirety of the internet. One extreme of the new freedom in IPv6 would be to drop all perimeter firewalling and firewall at each node. This strategy is likely a bad one but not entirely without merit. The slightly less extreme strategy of doing minimal filtering at the perimeter, purely packets that simply should not be heading into or out of any network. Things such as neighbour discovery, multicast listener discovery, Windows netbios protocols, and others. With the detailed firewalling moved to the node much of the firewalling work is widely distributed (assuming an attack didn’t focus on a single host, in that case, throttling could be done at the perimeter). One of the problems with this approach is the host firewall which would be easily torn down if the machine was compromised, inbound and outbound would be at the mercy of whoever is in control of that node. In my opinion this strategy should be at least experimented with and is probably suited to networks populated by geeks and datacenters.

On the other extreme is the heavily fortified perimeter, a well known paradigm to anyone who’s been involved in network security to any degree in the past decade. We block and/or control everything at the perimeter. SIP gateways, web proxies, stateful NAT firewalls. All this can still be accomplished under the new protocol short of hiding internal network addresses, however this would break many innovations that depend on peer to peer access. On a well configured enterprise network internet scope multicast streams may make it through but on smaller less throughly configured networks under this paradigm it may not, at present I don’t know enough about multicast routing and firewalling to say anything even remotely conclusive. The slightly less extreme version of this setup would be the good old stateful outbound only behaviour that exists in so many consumer NAT gateways.

There exist in between these two extreme ends of the firewalling world a range of happy mediums. One strategy would be to allow a certain range of ports though the perimeter for peer to peer applications such as the shared white-board application in Windows Vista. IPSec also requires end to end connectivity in a certain way and provides a secure and authenticated way to allow connections into the perimeter as IPsec can authenticate traffic and provide privacy as well. Microsoft has integrated IPSec into a remote access technology that works in a similar way to a VPN and yet subtly different, built entirely on IPv6 and IPSec technologies Microsoft’s Direct Access is an excellent example of a happy medium in the permissiveness of a perimeter.

At the end of the day it all depends on what you want from your network. A business will probably a strong perimeter, security oriented geeks could probably benefit from a node oriented firewalling strategy to make their projects and servers more flexible, and everyone will want IPsec authenticated/encrypted traffic.

Update: The Internet Architecture Board is considering IPv6 NAT in RFC 5902 in July 2010 and is discussed in a blogpost by Jeremy Stretch

Files: I’m releasing a script I wrote to generate ip6tables rules as part of this blogpost. It’s only v0.1 but I hope it will prove usable and useful to others. Suggestions and patches welcome. carbfw6-0.1.tar

Tags: , , , , , ,
Posted in Articles | No Comments »

Defcon 18 – photos and followup

Monday, August 2nd, 2010

Another year, another Defcon has passed. This year I went with CarbonLifeForm (who helped out with this post) and we attended talks, mingled with fellow enthusiasts, spread the psych0tik word, and enjoyed general debauchery. I was asked to take some photos for some friends who couldn’t attend this year, so I’ve pulled those together with some comments and added them to this post.

This year was quite busy, with far more people than have attended Defcon before. Unfortunately, that meant needing to wait in lines during other talks, so I’ll be looking forward to getting the video and being able to watch some of the ones I missed. A large number of attendees also made for a shortage of badges. This shortage resulted in a the “Lamer skybox party” Friday night for all those who didn’t get a proper badge. For a last minute party, the guys who put it together did a great job. I’ll be looking forward to hitting it up again next year.

Of the talks I saw, I particularly enjoyed the nmap scripting engine talk by Fyodor and the Razorback[2] debut by the Sourcefire guys. As always, Fyodor has been scanning the internet and this time he aimed nmap and a few nmap smb scripts at Microsoft’s net-blocks and showed us the results.

Razorback is Sourcefire’s new open source project, that’s built to correlate and process security events. It allows ambiguous data types and provides a framework for events to be dissected and analyzed by modules. Each piece of data is passed around to the various “Nuggets” for processing. It’s a neat system and I’ll be looking at writing some Nuggets to add to the farm.

In support of Gregory Evans and Ligatt, I wore a HackerGearOnline shirt on Friday. I was happy to see the con shirts labeled “Worlds #1 Hacker (conference)”, also paying due tribute to Mr. Evans. I didn’t manage to pick up one of the shirts, so I don’t have a picture of it.

Outside of being a bit over-populated, this year’s con was great. We enjoyed a number of interesting and entertaining talks and of course participated in the Defcon/Vegas shenanigans. Our flights were early and we missed closing ceremonies, but the announcement was made that Defcon would be moving from the Riviera to the Rio for Defcon 19[3].

MEDIA

This year’s badge was much more interactive than in years past. Including a mini-USB port and multiple challenges[1]

The following are a few shots of vintage hardware. This room (and a good bit of the hallway outside the room) had that “data center” smell. Directly below is a picture of the PDP-11[4].

A shot of the wall-o-sheep, as always, lots of great pwnage.

Hackbus – enough said.

Here’s a picture from Fyodor’s talk. If you can’t tell, this is nmap output for a Microsoft IP address.

Rocking my HackerGearOnline (order #19!) t-shirt. Thanks Greg!

As much as I try to keep this blog away from advertising, I’ve got to give it to IOActive for throwing a great party Saturday night.

A (now) empty bottle that ended up covered in Defcon18 stickers. The official booze of Defcon?

This one turned out to be off topic. I found this sticker on the dresser in my hotel room. I figured it was something con-related, but getting home and looking it up, it turns out to be a punk rock band. A bit of promotion for them too I suppose

REFERENCES

[1]Defcon 18 Badge – Ninja access bruteforcer

[2]Sourcefire VRT Labs – Razorback project

[3]Wikipedia – List of Defcon Venues

[4]Wikipedia – PDP-11

Tags: , , , , , , , , , , , , , ,
Posted in Articles | 1 Comment »

ESXi 4.0 – Management Methods

Sunday, March 21st, 2010

Introduction

Be it at work or at home, I spend a lot of time in virtual environments consisting mostly of ESXi. As part of the psych0tic network I run storm.psych0tik.net, an ESXi 4.0 box that’s currently running our IRC as well as the psych0tik GSProxy. Managing these environments is a task in itself, and I’ve found that no one tool gives me all of the functionality that I need. Additionally, I’ve found some confusion in what’s what as many of VMWare’s product’s have two names, due to a re-branding from Virtual Infrastructure 4 to vSphere. In this post, I’ll go through the various free management method’s that I’ve come across and use. I’ve also provided a short list of the non-free options. Listing all the features of each method would probably require a blog post or two per item, so I’ll go over some common uses as well as anything I think to be extra useful.

Embedded Web Server

The embedded web server provides a home page populated with links to various resources related to your ESXi 4.0 installation, the ability to browse your server’s datastores, and the ability to browse objects managed by the server. Accessing this service is simple and requires no setup. ESXi 4.0 installs with the VMWare ESX Server httpd and connecting requires nothing more than putting your server’s IP address into your browser (preceded by https://). The connection is protected by SSL (you may have to allow a self-signed certificate) and both the datastore and the objects are protected by basic HTTP authentication, using the same access controls as vSphere or the unsupported console.

vSphere Client (originally branded as the Virtual Infrastructure Client or VIC)

The VMWare vSphere Client provides the ability to monitor and manage multiple physical ESXi servers from the same console. This application runs on a separate machine, under Windows, and connects the the ESXi 4.0 host via the management network. Using the VI API, this tool connects to the hypervisor(s) on ports 443 and 903 and lets you manage the entire infrastructure from the hosts and switches to the server configuration.

For most of my day-to-day management the vSphere client is enough. You are able to start and stop virtual machines, take snapshots,  adjust resources, get performance graphs, spawn consoles on the various virtual machines, and adjust settings for the networking, guests, and host. In addition to the direct management of the guests, vSphere also provides an interface for working with your datastore(s) and allows you to upload/download files from the physical server as well as functionality for importing virtual machines.

Another advantage of the vSphere client is that it’s a standard VMWare product and therefore other products may be built with that in mind. Products like Cisco’s Nexus 1000v (not free), a replacement for ESXi’s native networking, integrates nicely with vSphere.

Maintenance Console/SSH

Sometimes I need to go beyond the high-level management of my servers. For example, sometimes I need to clone a virtual machine and, as far as I know, there isn’t a way to do this through the vSphere client . Tasks like this are better suited to the maintenance console, an unsupported command line interface for ESXi 4.0. As I mentioned, this method isn’t supported by VMWare and may void any sort of support you get. (I have a free license, I don’t expect support anyways.) The maintenance console is similar to the Service Console in ESX 3.5, but with reduced functionality. Using busybox, it’s possible to open this management method up to remote users as well as to allow SCP. (In my experience, vSphere’s upload/download is less stable than SCP.)

In the maintenance console, you’ll find access to the datastore as well as the raw virtual machine files. In addition to basic functionality, various VMWare specific tools have been added for manipulating guests or the hypervisor itself. vmkfstools, for instance, allows you to clone guess.

Also accessible through the maintenance console is the vSphere Interactive Shell (vsish) , another undocumented shell. From the bits of information I’ve found and some mucking around with vsish, it seems to be a mixture of an integration of the service console’s /proc nodes and linux’s mcelog, for polling reliability information. One interesting command I did find was to crash the ESXi VMM with a kernel panic: vsish -e set /reliability/crashMe/Panic

The Virtual Infrastructure metashell (vimsh) is yet another unsupported and undocumented shell accessible via the console or SSH. From what I can tell, after ESXi 3.5, this is only accessible through the non-interactive vim-cmd.

libvirt

libvirt isn’t solely meant for managing VMWare’s ESXi 4.0, rather it’s meant to be a toolkit to interact with various hypvervisors and their features. Because libvirt isn’t designed specifically for ESXi 4.0, there are functions of the API that it does not implement, such as managing vSwitches; however, it does provide a common programmable interface for managing in multiple languages. libvirt natively supports C/C++ and python and has bindings for Perl, Ruby, Java, C#, and OCaml. Beyond the limitations libvirt has in it’s ESX drivers, the free version of ESXi 4.0 restricts your access to read-only, disabling more of the functionality provided through libvirt.

virsh

virsh is the virtual shell that comes with libvirt as a method for managing domains under various hypervisors. Since virsh is built on libvirt, it also is unable to manage ESXi’s vSwitches and is restricted by the read-only nature of ESXi 4.0′s free version.

Among the usable features the virsh provides are the ability to list currently active domains, view information about the host’s and guests’ hardware and capabilities and view configurations. Unfortunately, from my testing it seems that a good majority of the more useful features virsh provides, like the ability to connect to a serial console, aren’t easily available (if at all) under the free version of ESXi 4.0.

NON-FREE

The management methods listed below are not free and therefore I have not been able to test them. Any functionality I’ve listed is based on what I’ve read. In most cases, you’ll be able to find a more in-depth description (read – “sales pitch”) in one of the references at the bottom of this post.

The vSphere Remote Command Line Interface (RCLI) provides the functionality that was previously available in the Service Console, specifically the esxcfg-* commands. The RCLI also allows the administrator to script various tasks. The esxcfg-* commands the RCLI relies on are still available via the unsupported maintenance console.

Functionality in libvirt and virsh seems to be increased through the use of a paid version of ESXi 4.0 as both are heavily restricted due to the read-only nature of the free version of the hypervisor. It appears that libvirt would gain a significant amount of functionality, including the ability to connect to a serial console, to manage domain’s state (start, stop, pause), and to edit configurations.

VMWare also offers the vSphere Management Assistant (vMA), a prepackaged linux VM to be used as a centralized management platform on which to deploy scripts and management applications.

Conclusion

ESXi 4.0 provides a variety of methods, free and licensed, to maintain and manage both the hypervisor and the virtual machines. The majority of my day-to-day tasks are covered in the free version of ESXi 4.0 and the tools available to it. The ability to manage domains, clone vmdk’s, retrieve configurations and device information, as well as to write applications based on both libvirt’s drivers and the VI API are all available for free. Paid options provide added function through the RCLI and tools like libvirt or virsh are made more useful. Beyond the obvious methods listed above, the undocumented and unsupported portions of ESXi 4.0 may provide additional functionality and features that will open new possibilities in the future.

Do you have another way to manage some portion of ESXi 4.0? Have I missed some serious piece of functionality in one of these items listed above? Let psych0tik know what you think in the comments!

[References]

libvirt website

Wikipedia entry on vSphere

Virtual Infrastructure API Reference

Accessing the ESXi maintenance console

VMWare ESXi SSH CLI Commands

Creating Kernel Panics with vsish

vimsh documentation

libvirt mailing list – questions (and answers) on libvirt’s ESX driver

vmkfstools man page

VMWare whitepaper on managing ESXi

VMWare vMA product page

[video] Cisco Nexus 1000v datasheet

Tags: , , , , , , , , , , , , , ,
Posted in Articles | No Comments »

Anonymous browsing or being a web ninja

Friday, January 29th, 2010

These days it seems like everyone is looking at what you do online. Online advertisers make money from the ads placed on your site, but they also gather statistics about all your visitors for their own purposes. Search engines store information for a variety of reasons. Social networks contain more information on our past than most of us can probably remember.  With the rise in popularity of the internet and it’s change from a place for geeks, hackers, and nerds to a place for the whole family it seems that the idea of anonymity online has almost disappeared. The internet has gone from the dark back-row in a movie theater to the digital equivalent of Orwell’s 1984.

I’ve always been skeptical about the idea of an intertwined real and internet life. When I was growing up and chatting to the sounds of a modem, I remember being told not to post my name online, not to tell people what school I went to, or my birthday. What are the first few questions when you register for Facebook? Who can see those? Obviously Facebook is a bit different than someone you meet in a chatroom, but it is interesting to think how freely we give out information we used to keep more protected. The reason richo set up the first psych0tik IRC server was that we didn’t like having conversations monitored by the guys running the messenger services. proxyElite was born from a desire to have reliable access to web proxies and aide in anonymous access. But I digress.

There are aspects of this problem that I simply don’t want to address. Social networks, photo-sharing sites, and blogging all seem to be inevitable parts of daily life. These are however known leakages. You don’t post to your blog or update your Facebook status with the idea that it’s just for your records. This is an issue as end-users we can do little more than educate ourselves on. Look at the Privacy Policy of websites and check your settings for options to disallow other users from viewing your content.

Beyond all these information giants are the internet’s motion detectors. Silent scripts and hidden images along with tracking cookies follow your movement, constantly reporting back to their creator’s servers. A recent post by the EFF shows that even your browser’s headers can be used as a unique identifier the majority of the time. [They provide a page that shows the entropy lost by each field checked, see the references below.]

In such a complex environment as the internet it can be quite difficult to tackle a problem as large as this, but with a mixture of configuration changes, Firefox Add-ons, and using proxy solutions it is possible to add to the difficulty of tracking your online activities.

Reducing and eliminating tracking cookies is a great place to start.

Configuring Firefox to delete private data when you close it is a great way to effortlessly limit the duration a tracking cookie is present. Many antivirus solutions also have an option to search for tracking cookies and remove them. This provides an external method to clear these, independent of the browser.

I use a multitude of Firefox Add-ons to not only protect my browser from malicious content, but also to help eliminate as many of the tracking technologies as possible. NoScript and Ghostery help to block scripts that might secretly send information back. RequestPolicy is great for defending against embedded tracking images (as well as CSRF); however, it is a bit over-zealous. User-Agent switcher allows me to adjust the entropy of some of my headers to be more “standard.”

Finally, using web proxies, Tor, or services like GoogleSharing it’s possible to cloak yourself even further. These services work to either distribute your connection across other machines or to reroute it through another. The Firefox TorButton Add-on also helps to mask your headers to make your session less unique. GoogleSharing is unique in that it doesn’t proxy all requests. Rather, only requests made to non-authenticated Google services like Google’s search. The requests are routed through a GoogleSharing server (via a Firefox Add-on), normalized, and passed through. Other users using the same proxy would add to the terms and add more chaff to deter monitoring by Google.

The Frankenstein of security that is now my browser with these assorted Add-ons and changes isn’t exactly as fluid or functional as before. A lot of sites break until I sort out which Add-on has blocked which critical script. Proxies and Tor make requests run more slowly. The web is definitely a more complicated place, but isn’t the effort worth it? You wouldn’t casually walk down a dark alley filled with dangerous looking folk without some protection. All the family friendly, Web 2.0 sites and services make the internet look like a lovely place, but let’s not forget that the dark alleys exist and are often in between all the “safe havens” we travel to.

References:

EFF’s Panopticlick Research Project on Determining Browser Entropy

EFF Blog on User Tracking on the Modern Web Part2 Part3

GoogleSharing Firefox Add-on

Ghostery Firefox Add-on

Mozilla Support on Clearing Private Data

Tags: , , , , , , , ,
Posted in Articles | No Comments »