10 Year Anniversary!

Ten years ago today, I released my first blog post on SamuraiNet and began my career as a blogger. Over the years there have been many changes, ranging from content types (or lack there of) to the merge with Psych0tik and the creation of the Psych0tik Network. It has been a long and wild ride, but I’ve definitely learned a few things along the way. Despite the occasional gaps in content, this blog isn’t going anywhere soon so keep checking back for new research, blog posts, and news.

Here’s looking forward to the second decade of our adventures together!

-samurai

Posted in psych0tik News, SamuraiNet Archive | Tagged , , | Leave a comment

The psych0tik domain parking host

Over the years we at the psych0tik network have amassed a number of domains.  Some of these, such as proxyElite.net or samurainet.org, were previously used to provide our services, while other domains have never seen real use.  Like the proper techno-hoarders we are – we have refused to give up even the completely unused domains.

In an attempt to get some value out of these useless domains, I’ve set up one of our less active hosts to act as a domain parking IP address[1]. We will be using this IP address to host very basic advertising content, in an attempt to generate some value from the domains we refuse to give up.

The psych0tik network has never, and will never, show advertisements or 3rd party content on our public services – and we don’t intend to start now.  Instead, we will be pointing these domains at our new domain parking server and hosting advertising content exclusively on that host, to avoid including potentially undesirable content on any psych0tik services.  Any income generated from these services, or our other “support psych0tik” options, will be used to offset our operating costs or expand our operations.

[1] Domain parking hosts are typically used by registrars or other wholesalers to redirect any unsold domains to a single IP address.  This IP address usually shows advertisements for either the registrar or affiliates.

 

Posted in psych0tik News | Tagged , | Leave a comment

Automating Security Analysis

A few months ago I began working on a short paper to generate CPEs used for renewing my GIAC/SANS certifications.  With few requirements around what the content or format of the paper had to be, this left me opened to writing about something I truly enjoy – automation.

For years I have worked on IRC bots[1], proxy monitoring[2] and wargame[3] automation in my spare time.  At work I spend the majority of my time designing, implementing and managing automation for a variety of purposes.  This paper is not meant as a definitive guide on all things automation, but rather as my perspective on designing automation specific to analytical workflows.

The final version of this paper is a long way from where I started, and almost double the length required for my certifications.  I ended up learning quite a lot about something I already thought I knew in the process.  Many thanks to those of you who provided feedback and review of this to help me get it polished.

Hopefully this is useful or interesting to someone other than me: Automating Security Analysis

[1] http://blog.psych0tik.net/2010/10/writing-irc-bots-using-ii/

[2] http://blog.psych0tik.net/2011/10/updates-to-proxyelite-pe-api-and-fresher-proxies/

[3] http://overthewire.org/warzone/

Posted in Articles, Guides | Tagged , , , | Leave a comment

pwning bin2json

So OJ posted this pwnable that had evidently survived the
BsidesCBR and Kiwicon CTFs and it got my interest.

I haven’t really been CTFing actively for years, but I still occasionally enjoy
pwning things, and I don’t remember if I’ve ever done one of OJ’s challenges.
So I grabbed a copy, and decided to see how lazy I could be while still shaking
a shell out of this thing.

His challenge talks some goofy binary protocol that I didn’t even attempt to
understand, but he helpfully included a sample script that emits a valid
message. I started building qemu for afl, and had a quick look at the
binary in binja. A quick patch to make it exit after a call to display_all
later to make fuzzing practical later, I started the fuzzer and got dinner.

Just to be clear, the process of setting up afl is really nothing special. All I did here was put a single testcase (The verbatim output of OJs sample) into an input directory and run it with:

afl-fuzz -Q -i input -o output -- ./bin2json

After about 20 mins I had a crash, and a quick inspection revealed this:

0x0000000000400ca3         mov        rax, qword [ds:rax+0x210]
0x0000000000400caa         test       rax, rax
0x0000000000400cad         je         0x400cc3

0x0000000000400caf         mov        rax, qword [ds:_IO_stdout]
0x0000000000400cb6         mov        rsi, rax                                  ; argument #2 for method fputc
0x0000000000400cb9         mov        edi, 0x2c                                 ; argument #1 for method fputc
0x0000000000400cbe         call       fputc

0x0000000000400cc3         mov        rax, qword [ss:rbp+var_338]               ; XREF=display_all+100
0x0000000000400cca         mov        eax, dword [ds:rax+0x200]
0x0000000000400cd0         test       eax, eax
0x0000000000400cd2         jne        0x400ce1

0x0000000000400cd4         mov        qword [ss:rbp+display_func], 0x400b9d
0x0000000000400cdf         jmp        0x400d1d

0x0000000000400ce1         mov        rax, qword [ss:rbp+var_338]               ; XREF=display_all+137
0x0000000000400ce8         mov        eax, dword [ds:rax+0x200]
0x0000000000400cee         cmp        eax, 0x2
0x0000000000400cf1         jne        0x400d00

0x0000000000400cf3         mov        qword [ss:rbp+display_func], 0x400bfa
0x0000000000400cfe         jmp        0x400d1d

0x0000000000400d00         mov        rax, qword [ss:rbp+var_338]               ; XREF=display_all+168
0x0000000000400d07         mov        eax, dword [ds:rax+0x200]
0x0000000000400d0d         cmp        eax, 0x1
0x0000000000400d10         jne        0x400d1d

0x0000000000400d12         mov        qword [ss:rbp+display_func], 0x400b4e

0x0000000000400d1d         mov        rax, qword [ss:rbp+display_func]          ; XREF=display_all+150, display_all+181, display_all+199
0x0000000000400d24         lea        rdx, qword [ss:rbp+var_320]
0x0000000000400d2b         mov        rdi, rdx
0x0000000000400d2e         call       rax

So, a classic uninitialized variable bug. var_320 is not initialized on
entry, so if var_338 is not 0, 1 or 2, then the value is left
unchanged, and subsequently called.

This is pretty simple to exploit, just find some other function called by main,
and corrupt that same slot on the stack, then arrange for display_all to be
called, and control pc.

As it happens, OJ thought of this and arranged the stack neatly in the only
other callable function (create_person) to avoid this. This is where I got
bored and wandered off, since this testcase gave me nothing to go on.

A few hours later, afl had found a second testcase that allowed PC control. I
never actually got around to reversing it to see how it worked, but quickly
looking at the crash I saw pc = 0xfcfcfcfcfcfcfcfc, and my testcase had a few
massive slabs of fc bytes in it.

A quick script to tamper with them in turn until I found which bytes were
corrupting that slot on the stack revealed that I had control of about 440
bytes of stack. I put a few quick gadgets together by hand to pop over the
garbage I needed on the stack, and then used ROPgadget to
generate a chain to get a shell. Their chain was gigantic, I dicked around for
a while trying to find a couple of neat gadgets to smuggle r10 into rax, which
was pretty close, and then remembered that I had full control just used a pop
rax
to populate the syscall number, which ROPgadget doesn’t do for reasons I
don’t understand.

Finally:

(yes, my dumb DO image I brought up quickly doesn’t have lldb in it’s repo for
some reason)

(gdb) r < /root/pwn/exploit_with_payload
Starting program: /root/pwn/bin2json < /root/pwn//exploit_with_payload
process 2099 is executing new program: /bin/dash

You can find the shitty script I used to patch my testcase here:

In the spirit of really, really avoiding investing effort in this, I’ve left
all my janky experiments in as comments.

It’s worth shouting out that voltron saved my ass again, the stack view is the best when you’re trying to map between a binary testcase and where things end up in memory.

And of course, thanks OJ for nerdsniping me!

Posted in ctf | Leave a comment

IRC outage (2015-04-15)

Through a ridiculous series of events and incredibly poor luck (read: I hope it’s not incompetence) our VPS provider is experiencing power issues. You can follow along at TransIP’s , some of you may need to hunt down a babelfish.

Updates will be available through this post and through the psych0tik twitter.

Since I can’t make it go, I’m going to take a shower instead.

Update@2015-04-15 03:11 UTC:
Some systems are returning to service but we don’t have VPS control panel yet. I might recommend a good pair of jumper cables to them soon.

Update@2015-04-15 05:26 UTC:
Refresh; still nothing; repeat. I might sleep first and get everything running first thing in the morning. I’ll give them another 30 minutes.

Update@2015-04-15 06:19 UTC:
IRC is back!

Posted in psych0tik News | Leave a comment

QuickBlog: Trac, SVN, Python and threads

It’s been a long time since I’ve done any reasonable blogging.  Between work and life, my plate has be been full and the last thing on my list usually is “write a blog about stuff you’ve done.”  More times than not, the things I’ve done I cannot write about due to work contractual concerns with work.  In an attempt to revitalize some posting and encourage myself to push more content out, I’m going to try to start creating these QuickBlogs.  These will be mostly focused around specific, bite-size content and will likely get a lot less word-smithing than my typical articles.  So there’s the intro.  Quick. Easy.  Let’s move on to the juicy stuff.

The problem

As part of my job, I’m often forced to look at or work on obscure setups that probably shouldn’t be production solutions .. and then fix them.  In this case, we’re talking about a Trac instance supporting an SVN repository.  Trac is being run under Apache2 using mod_wsgi in it’s daemonized, workermode configuration.  The problem statement is simple:  Trac begins spitting out 500 response codes to certain requests and before long, all requests are returning 500’s after exceptionally long wait times.  To make matters a bit more interesting, these errors began occurring mere hours after a full system upgrade.

The first problem was logging.  Trac’s logs simply didn’t tell us what was happening.  Apache’s logs were even worse, random 500’s with no real indicator or commonality between the failures.  A peek at the resources on the box showed CPU and memory consumption would eventually reach obscene levels and the only obvious fix required processes to be killed.

After a number of eyes spent time pouring through the logs we identified a particular SVN revision that seemed to be giving Trac the business more than others.  A quick check of this changeset revealed a 1.2 Gigabyte diff of a large file with significant modifications.  This changeset also was committed just a few hours after our system upgrade, making sense with our problem timeline.  Now that we’ve got a potential root cause, it was time to sort out this logging issue.

Trac’s native logging leaves something to be desired, but the trac.log file did include enough log text to make finding the last logged action within the code trivial.  Looking around within the code, I noticed that Trac provides an object for doing debugging logging that I could use to dump to the trac.log file.  Most code within Trac has access to the self.log.debug() function, which I made heavy use of throughout to identify where execution was going, what variables looked like, and most importantly – where the code eventually would block and stop processing the diff.

A few hours of debugging later, I had my code location (or at least close enough to know what the issue was.)  After the diff() was run within Trac, Trac attempts to htmlify() the results for presentation to the user; an action that takes far too much time when your diff is 1.2 Gigabytes.  With the additional logging I added, I could see the 500’s and the completion of this long-running function coincide, telling me that I’ve found my bottleneck.  This is only half the answer however; as we have pages that have nothing to do with this, or any, changeset to the code that also will toss 500’s back at us.

The obvious question was “is this related to our massive changeset issue,” to which I thought an equally obvious answer was “probably, yes.”  During the troubleshooting of the changeset issue, I noticed that the logging and seemed to lag behind my actual work.  I’d make an update to the code, refresh the Trac web page, and my logs would not reflect my code changes.  A few requests later, I’d see those changes.  Based on this, I was fairly certain that something about these diffs was cascading into other execution.

With a problem definition in mind, I moved forward to test my problem hypothesis.  I modified my Trac logging so that I could clearly see when requests came in and completed.  I then opened up a two different pages in the Trac web interface.  One page was a diff-view of the problem changeset and the other was an unrelated, lightweight page that should have always returned content.  I issued a few fast paced requests to the diff-page (literally just spam clicking the refresh/cancel button in firefox), followed by a single request to our canary page (in this case a random Trac ticket.)  Watching the logs, I saw exactly what I expected – when each diff-page request got to the htmlify() function, it would block ALL THE OTHER REQUESTS from even being accepted (at the mod_wsgi level, Apache still saw them.)  As each request finished, the next in queue would be processed and would block again, and again and again.  This cascade of blocks caused the requests for our ticket to fail because even tho the Python/Trac code executed quickly, by the time the request was even processed by Trac, Apache had already given up on it and moved on with a 500.

So the question you may be asking here is (especially if you don’t know me, and haven’t heard a rant on python threading) – Why didn’t a worker-mode, daemonized version of mod_wsgi simply allow the various threads to handle the requests in parallel, completely side-stepping this problem?  The answer is within the Python interpreter itself and it’s GIL (as well as potentially the implementation on the Trac side.)

Python’s GIL acts as a safety mechanism, preventing you from doing “unsafe things” to volatile shared memory.  This works excellently for protecting your data, but also means that Python’s threads can only run one at a time, per core/processor.  Having only a single instance of Python means that, because Python can only use a single core/processor per instance, every request blocked the rest.  This resulted in a single bad page being able to push the entire request stack so far back in time that they all had to result in 500’s.

While Trac could be patched to detect large diffs to ignore or the Apache/mod_wsgi set-up could potentially be moved to a pre-fork/embedded mode (much more memory intensive, but less CPU blocking) – we determined the best fix was to simply purge our SVN of this bad revision (which really shouldn’t have been committed to begin with.)

Fixing the SVN was fairly trivial.  I identified the problem revisions, and used svnadmin’s dump and load, with a few flags, to rebuild the SVN repository minus the bad changes.  Note, this may cause problems if you’re removing anything that is modified by a future revision, in my case I knew these weren’t further modified and were safe to remove revisions of.   We’ll assume an SVN repository going up to revision 100, with problem revisions 89 and 92:

  
  mkdir /tmp/svn-workspace
  svnadmin create /tmp/svn-workspace/fixed-svn
  svnadmin dump /path/to/svn -r 1:88  > /tmp/svn-workspace/first_part.dump
  svnadmin dump /path/to/svn --incremental -r 89:91 > /tmp/svn-workspace/second_part.dump
  svnadmin dump /path/to/svn --incremental -r 93:HEAD > /tmp/svn-workspace/third_part.dump
  svnadmin load --force-uuid /tmp/svn-workspace/fixed-svn < /tmp/svn-workspace/first_part.dump
  svnadmin load --force-uuid /tmp/svn-workspace/fixed-svn < /tmp/svn-workspace/second_part.dump
  svnadmin load --force-uuid /tmp/svn-workspace/fixed-svn < /tmp/svn-workspace/third_part.dump
  trac-admin /path/to/tracdata/ repository resync YourSVNName

References:

http://blog.dscpl.com.au/2009/03/load-spikes-and-excessive-memory-usage.html

Posted in QuickBlogs | Tagged , , , , , | Leave a comment