In the part 6 of the series I have already explained the topic of the coverage guided fuzzing and there is little point of repeating that. Still, as a brief reminder - we will be using the 64 Kb of shared memory to track all the branches we are hitting. My original approach of incremental counter acting as an index into the bitmap will work for most of the cases because most program tend to stay south of 15,000 branches. If however we end up fuzzing a program that has more than 65,000 branches we will be facing a problem that Terry Pratchett once described as trying to pull a rabbit, size 7 from the hat, size 4.

Before we move further we need to talk what kind of characteristic the good coverage bitmap should exhibit to be well suited for our purpose - accurately tracking the given program execution flow. Among the most important one is the good accuracy, low collision rate, good density and, because we are talking about fuzzing - good performance. We will address those elements as we are moving along with the article.

Before our attempt to design a new system let’s first try to understand how AFL is doing that. The solution is relatively simple (but elegant). Each basic block is assigned a random number - mostly to provide uniform distribution and avoid clustering. While transitioning from block A to block B the corresponding numbers are XORed together and the result is used as a index into the coverage bitmap. With just that approach we would end up with a situation where not only we can’t tell direction of the transition but also all self referencing blocks (or tight loops) would end up pointing at the same index of the bitmap. To solve that AFL replaces the number associated with the previous block using the bit-shifted number associated with the current basic block.

During the development phase of the AFL clang was not supporting the proper coverage interface. Implementing our fuzzer roughly 10 years later has certain advantages. The first one being - we can use SanitizerCoverage instrumentation that already operate on branches, so we do not need to XOR the basic block identifiers. We can simply assign a unique number to each branch and track the hit on our coverage bitmap. That still leaves us with the problem of the bitmap size and using index outside of range.

First possible solution is simple - we can assign numbers incrementally and just do the modulo 65536. This will leave us with two problems - structural clustering and the preservation of lower-bit patterns. We can demonstrate the problem with the image.

This is a 256 by 256 bitmap that shows our coverage trace. Each branch index is translated into a two dimensions and each hit is drawn with a different color. Simulating the program execution we can clearly see the clusters of values. The effect here is very visible and shows that results are clustered with upper and lower parts of the image populated very sparsely. If you think this looks like a Gaussian distribution you are absolutely right.

The reason for that is me cheating a little bit. While I was creating a visualization I didn’t wired it up with a real program - instead I have generated 15000 random numbers with the Gaussian distribution and the modulo operation preserved it.

Another example is the bucketing - instead of doing 1:1 mapping we treat every index of the bitmap as a bucket for a group of values. Simply shifting the obtained value 4 bits to the right we get the 1:16 ration so we can fuzz even the most demanding projects. With that we however face a rather serious case of result clustering. We risk many branches ending up in the same bucket skewing our results. Just look at the image.

There is a third solution that borrows one element from one famous Italian mathematician. We will be using the Fibonacci golden ration constant that, expressed in hexadecimal format, equals 0x9E3779B9. You can and should read more about it in more reputable sources but, briefly, this number has one important characteristic - great scattering property. If we do multiply incremental numbers by the golden ration the results will be spread around the given space and every consecutive multiplication divides the larges remaining space of the bitmap in half. Image below demonstrates the effect of our new method.

Even with the numbers with the Gaussian distribution all the branches are evenly distributed across our plane.

Now, the code responsible for that is quite simple

#define GOLDEN_RATIO_32 0x9E3779B9

void __sanitizer_cov_trace_pc_guard_init(uint32_t *start, uint32_t *stop) {
  if (start == stop || *start) return; // Prevent double-initialization

  uint32_t unique_id = 1;
  
  // Assign a pre-computed 16-bit bitmap index to every single guard slot
  for (uint32_t *x = start; x < stop; x++) {

    uint16_t bitmap_index = (unique_id * GOLDEN_RATIO_32) >> 16;
    *x = bitmap_index; 
    
    unique_id++;
  }
}

Best part is - multiplication with wrapping is very fast on modern CPU (just few cycles, and often can be pipelined) and we can compute all the branch indexes just once, during the program initialization. Now, all that remains is to use the __sanitizer_cov_trace_pc_guard(uint32_t *guard) function to properly increment the value inside a given shared memory space, so at the end of the run we have the accurate representation of a trace.

When it comes to bitmap comparison is, again quite fast. In C we would simply write it like this:

memcmp(run1_bitmap, run2_bitmap, 65536) == 0`

Also, because the bitmap are of the same size and can be aligned in memory the compiler can turn this into super fast SIMD operations. My only issue with that approach is that we get a binary response - the coverage is either identical or it is now. Some time ago a friend of mine introduced me to the concept of location-sensitive hashing. In theory - and I am saying that only because I haven’t write the program yet - we can compute two hashes and calculate the Levenshtein distance between them. That would tell us how very different were the two runs we just observed. With that approach maybe we can actually capture the outlier runs and new branches bit better. Maybe that would be a good topic for the next article.

Problem to solve

Let’s start with a very simple multi-threaded server written using the tokio framework.

#[tokio::main]
async fn main() {
  let server = UnixListener::bind(SOCK_ADDR).expect("Error binding socket to file");
  loop {
    let (sock, _) = server.accept().await.unwrap();
    
    tokio::spawn(handle_connection(sock));
  }
}

Obviously, what I am taking for granted here is that you, the reader, are familiar with that framework. But even if you are not, the code is fairly easy to understand. You create a socket (in this case this is a Unix Socket, created by the UnixListener::bind() call), accept incoming connections by calling the server.accept() and hand the socket over to a newly spawned thread.

Shared state

If you are writing an echo server you don’t need to synchronize anything. You just read from the socket and you write back to it. If however you need to hold state between different callers you have to come up with something. People who have learned concurrent programming with C or C++ will most likely gravitate towards mutexes. This is fine, Rust has you covered with the Arc<Mutex<T>> construct. Simply wrap your favorite data structure and lock when trying to write to it. There are however some small problems with that approach. While Rust can be memory-safe it is definitely not deadlock-free - and in the complex programs you can get into deadlock pretty easily - try holding a lock across multiple await calls and see what happens. Even if you are able to avoid deadlocks there is a chance you will make the program much slower. You can also design a lock-free structures, but that won’t be fast.

There is one approach that is a little bit better - channels. To demonstrate how this might work we make small modifications to the previous program.

async fn main() {
  // Create a channel for communication
  let (tx, rx) = mpsc::channel::<IPCMsg>(64);
  
  // spawn process for synchronization
  tokio::spawn(data_broker(rx));

  [..]
  tokio::spawn(handle_connection(sock, tx.clone()));

First of all, we are creating two halves of a multi-producer, single-consumer channel. The first half, denoted as rx, is passed to our thread that will be responsible for data synchronization by receiving IPCMsg from all the threads handling user connections. Those other threads will take the other half of the channel denoted as tx. Because the channel supports multiple producers, we can clone the tx half and give one to each thread.

Now the synchronization function can read the messages sequentially and do whatever needs to be done with the incoming data - write it to a database for example. Like this:

async fn data_broker(mut rx: mpsc::Receiver<IPCMsg>) {
  while let Some(msg) = rx.recv().await {
    // process data
  }
}

Writing back

There is one crucial thing we have overlooked so far - how do we actually tell the thread handling user connections that we are done with data processing. Looks like we are missing the channel. To explain how we are going to do that we start by showing the exact definition of the IPCMsg enum.

enum IPCMsg {
  SEND(oneshot::Sender<IPCMsg>),
  RECV,
}

The interesting part here is the value being held by the SEND variant of this enum - a half of the one-shot channel (that, as the name suggests, we send for transmitting a one-of message). Now, every time the connection-handling thread feels a need to communicate with a data broker it creates a one-off channel and sends a transmitting half along the message while listening for the response. This might look like this:

  let (otx, orx) = oneshot::channel::<IPCMsg>();
  tx.send(IPCMsg::SEND(otx)).await.unwrap();

  match orx.await {
    Err(e) => eprintln!("Error: {}", e.to_string()),
    Ok(val) => println!("Response: {:?}", val)
  }

Summary

The code here might be fairly simple but it has solved some of my problems while engineering multi-threaded applications. This pattern is so effective that I also applied it in another program I wrote, which was in Go.

Reluctantly I have to agree with that statement with one, very strong caveat: there is work and there is work. I too enjoy something that is commonly referred to as deep work. Obviously every one of us has a different endurance and mental resilience but working for 10-12 hours in a day is completely possible. I know because I’ve done it. Not for a prolonged periods of time, but in shorts sprints of course. If you can focus on a computer game for that amount of time you can also work that long. The deep work can be very satisfying. In the movie The Internship (let’s disregard that it had exactly zero overlap with reality) there is a scene where one of the characters (played by Vince Vaughn) asks “Do you remember what ii felt like being that good at something?”. That is the feeling that deep work can give you. And you can draw immense satisfaction from it. So immense that you don’t even count hours anymore. So yes, in this case Sergey is right. When you hunt for bugs, develop an exploit or code new systems then you need those long hours to load the context into your brain and gain the productivity you want to see.

There is, however, another kind of work. The one that working at Google also made me acutely aware of. It is a work of constant interruptions and context switching. The work of tactical objectives, quashing fires, answering questions and acting as catch-all rule for anything security related. But at least that is work. There are also meetings, mandatory trainings, performance reviews and other time consuming events. That kind of work gives very little satisfaction, is very exhausting and mentally taxing. If you spend more than few hours in a day doing that you will have zero capacity for any kind of deep work later in a day. Still this kind of work needs to be done, so if you are not doing it someone else will have to. Sure - the higher level you have the bigger amount of that you can delegate. I am sure that Sergey is not randomly pulled into a meeting because compliance team suddenly needs to understand what is the difference between 7.9 and 8.0 CVSS score.

We have also glossed over other aspects of your life. Some of us have kids, families or people that depend on us. Sure, 10 hours in a day of deep work would be great, but we have groceries to do, kids to pick up, doctor visits to attend. Money of course can solve some of those problems. If you wake up in the morning and your breakfast is ready, your clothes are neatly folded and ready to wear, kids are being chauffeured to school and you don’t have to worry about cleaning up, you can devote your full energy to work. Of course, not all of us are in this position (not to mention - you definitely should not outsource the raising up your kids part).

I am not overly worried about Google. I like to think that Sergey is a smart man who can understand those limitations. After all, a lot of teams at Google have delivered amazing products without toil and death marches (like Chrome) but this is a very different topic. What I am worried about are countless followers and pundits who will take the 60 hours thing as gospel and try to apply it indiscriminately to any breathing body around them. With disastrous results.

Background

How did we even end up here and why would we even translate structures between C and Rust? When you do any kind of system programming you will be talking to a kernel. A lot. Sadly you can’t do it over Protocol Buffers. Or even Cap’n’Proto. Or, thank god for that, over JSON and XML. You often have to send a pointer to a chunk of memory through some syscall or ioctl. Examples? Here you are..

  kvm->mem.slot = 0;
  kvm->mem.guest_phys_addr = 0;
  kvm->mem.memory_size = kvm->ram_size;
  kvm->mem.userspace_addr = kvm->ram_start;

  ret = ioctl(kvm->vm_fd, KVM_SET_USER_MEMORY_REGION, &(kvm->mem));

This is the code that initializes the KVM memory region. We just initialize several fields in the kvm structure and pass the pointer via the ioctl call to the kernel. Simple, right? We should be able to just create a structure in Rust, call the same ioctl and be done with it. Sadly, this won’t work. To understand why, let’s inspect how the structures look in memory.

Memory inspector

First of all, we are going to need a simple program that we can use an example.

#include <stdint.h>

struct x {
  uint16_t field_1;
  uint32_t field_2;
  uint16_t field_3;
};

int main(int argc, char *argv[]) {
  struct x variant;
  variant.field_1 = 0x4141;
  variant.field_2 = 0x42424242;
  variant.field_3 = 0x4343;


  return 0;
}

Let’s cover what is going on - mostly for posterity. Our structure have three fields that have, respectively 16, 32 and again 16 bits. We fill those fields with unique values so it is easy to distinguish them while looking at memory dump.

Checking them under gdb yields following result:

pwndbg> x/4x &variant
0x7fffffffdb74: 0x00004141      0x42424242      0x00004343      0xffffdcb8

We can see that while the field_2 occupies 4 bytes (32 bits) as we have requested there is some nasty looking padding in the form of 0 around the values we clearly wanted to be only 2 bytes wide. The reason for that is that certain C standards (like C99) requires two things - structure fields appear in the same order they were declared and addresses of the fields are aligned to 4 bytes. This is why you see those nasty gaps between them.

Let’s see what is the Rust opinion about said standard. As we’ve done previously - let us write a simple program, but this time in Rust.

#[derive(Debug)]
struct X {
    field1: u16,
    field2: u32,
    field3: u16,
}

fn main() {
    let variant: X = X {
        field1: 0x4141,
        field2: 0x42424242,
        field3: 0x4343,
    };

    println!("{:?}", variant);
}

We don’t need to explain too much as the program works exactly the same as the previous one. If we look at it in the debugger the results will be bit different.

Protip: break variant_rust::main will get you to your main function. Obviously replace variant_rust with the name of your program

pwndbg> x/4x $rsp
0x7fffffffd8e0: 0x42424242      0x43434141      0x555a9b10      0x00005555

This definitely doesn’t look like the C structure. The positive part is that the structure takes less space in memory because it is packed and some fields were rearranged. Obviously Rust compiler can handle that but trying to pass this structure through the FFI boundary is no bueno. And if you do, I guarantee some long hours getting very intimate with the debugger trying to figure out why things have just exploded.

So what is the solution? Fortunately Rust has a directive just for that - #[repr(C)]. This will ensure that the resulting structure will be compatible with the C layout, fields won’t be rearranged and the correct padding will be used. Use it like this:

#[derive(Debug)]
#[repr(C)]
struct X {
    field1: u16,
    field2: u32,
    field3: u16,
}

There are of course other fun aspects to handle - data type width (int should be i32 but on some old systems it might as well be i16), pointers, enums and other fun elements. Read the documentation please. I will write a bit more about this in the next article.

Ah, right, because I have not mentioned that in the beginning - I am writing a short series of articles about writing your own Virtual Machine Manager using KVM and I have decided to do it in Rust. Stay tuned.

There are many schools of handling this - some languages use exceptions. Other, like Go return value and error separately and it’s the user job to handle it correctly. The smart Go programmer always have a macro bound to one of the keys to immediately output if err != nil because this is the most commonly typed line in many go programs. This approach however has one main problem - developer can still ignore or forget the returned error and continue with the flow like nothing has happened.

Rust, like many functional languages (and, some might be surprised - the Google style of C++) is using something called monadic types to bind return value and the error into one type. In simple terms - many functions in Rust return a Result<V, E> - an enum that must be “unpacked” before a value can be used. The full definition of said enum looks like this:

enum Result<T, E> { 
	Ok(T), 
	Err(E) 
}

First time you encounter a situation where you cannot simply use the value that the function just returned is quite a teachable moment. The Rust book initially give you just two tools to handle that situation - .unwrap() and .expect(). As the name suggest they simply unwrap the value out of the container and allow you to use it. The .expect() allows you additionally specify the error message that will be displayed upon hitting that particular function. You might ask where exactly this message will be displayed and it is time I mention one important characteristics of the aforementioned methods. They make every error irrecoverable. Upon the call your program will just terminate.

Program stopping every time you encounter a tiniest of error might not be ideal but actually in many situations this might be desirable. First of all - you might want to treat certain errors as the end of the road for the program. Also - when you just prototype some functionality you don’t want to be bothered with some complex error handling routines.

When you learn a bit more about the Rust you will realize that, given we are working with the enum we might just use match to get some more flexibility. Code below demonstrates this:

match function() {
  Ok(v) => // handle normal situation,
  Error(e) => // handle error
}

The match expression forcing us to actually handle all enum cases makes sure we handle all the errors we encounter. We can even be smarter about it - if a function returns different kinds of errors (like File::open()) we can treat all of them separately by adding more arms to the expression.

The most common situation in Rust happens when we want to assigned a returned value in case of success and return an error in case we have encountered one. We can express this in code like this:

let something = match function() {
  Ok(value) => value,
  Error(error) => return Error(error)
}

This situation is actually so common that rust developers have decided to save us all some time typing this elaborate match and you can just replace it with one character added at the end of the function call - ?. I think this is the best invention since keyboard shortcuts. That also helps you just to propagate all kinds of Errors up the call stack so you can handle them all at the level where you feel comfortable.

So far we haven’t learned anything that is not already covered by the Rust book and you might be slowly losing your patience. Trust me - this lengthy introduction was needed.

The reality of programming is that very often you won’t care about one or the other arm when handling the Result. For example - you have a function that change the internal state of some object - there will be no value to return but you still need to handle the error - for logging if nothing else. This can be done by a following pattern:

if let Err(e) = some_call() {
  // log error
}

Of course a opposite situation is also possible - it is less common but we might want to handle the situation where we might want to ignore the error and just get the value.

if let Ok(v) = some_call() {
  // log success
}

Truth be told the if let constructs is much more commonly used when handling the Option<T> type.

Rust has not shortage of tools that we might want to use when dealing with errors - one I am commonly using is the .map_err(|e| ... ) expression. Typically, when writing a program that touches many different aspects or domains you want to make the error handling fairly uniform. The system functions will often not obey - each one of them will return different set of errors - IO ones, Network ones etc. When calling those functions you might just want to map them to your own type for easier handling later on.

Another interesting way is to defer to closures - not always you want to handle the error by writing an explicit match. Sometimes a .unwrap_or_else(|err| ...) might be exactly what you will need to simply log and bail out. Or, even better, provide a default value in the absence of the real one.

The great thing about those functions operating on Result and returning yet another result is that you can chain them to achieve the desired result. One thing is for sure - whatever scenario you might come up with Rust has you covered.

Originally this was just a series of separate notes on my Mastodon profile. I’ve decided, in case the site disappear, to grant them more permanent form and post them on this blog.

For the past 10 years I’ve been going to Bay Area at least 4-5 times a year so I’ve manged to know the area pretty well. At first I was typically staying in Santa Clara but later on I have migrated near Castro Street in Mountain View.

One of the places that I’ve particularly enjoyed was “Book Buyers” - a bookshop right in the middle of Castro Street. It specialized in used and out-of-print books and I’ve sniped so many great samples there. My fondest memory was when I just entered with a title of a story I have read many years ago in a “Nowa Fantastyka” - Polish magazine for sf&f fans. The salesperson there was extremely helpful and we have managed to locate a book that included that particular story I was looking for. For those interested - it was “Lifeboat on a burning sea” by Bruce Holland Rogers. When I heard the bookstore was forced to move out I was crestfallen.

I’ve never managed to visit them in new location in Gilroy. They have survived another 6 years there until they had to move once again. This caused the owner decision to shut it down. I will miss them.

The Fish Market was the first restaurant I’ve visited in after coming to California. It wasn’t a very conscious choice but rather a matter of convenience - I was staying at The Domain Hotel at the time and it was the closest restaurant I saw on google maps.

It turned out to be a great choice - food was great - especially if you like the fresh fish straight from the wood-burning grill. It this was also the first time I was truly exposed to American portion sizes - I had a smoked fish sampler and the glass of wine (it might have been from Francis Ford Coppola vineyard) and I was done. Nevertheless, the entire experience was great and I’ve been returning there for many times. Some memorable dinners there.

Sadly, last year they were forced to close all their locations in the Bay Area.

The pop-culture wisdom says that steak is the national dish in the US and the Bay Area has no shortage of good places that server one. If you feel fancy you can of course visit the Alexander Steakhouse and eat 3 oz slice of Kobe beef that costs you an arm and a leg. Some of my friends fall for that and they were not particularly thrilled by the experience and even less about the final bill.

To avoid surprises like that I always try to go be the recommendation of the people living there. One evening my friend Luca took me to Black Angus in Sunnyvale and I have enjoyed it a lot. While I do not appreciate the barn look the food was making me forget all about decor and focus on eating. We even had one or two team dinners there. Sadly, after almost 40 years the restaurant was forced to close down

The Prolific Oven was my go-to breakfast place. They had two locations close to Google apartments - in Sunnyvale on Washington Avenue and in Santa Clara near Moreland Way. While they specialized in cakes and other sweets I really enjoyed their benedicts and omelets. The prolific oven closed in 2019, after 39 years of presence in Bay Area.

Niji Sushi near Castro Street in Mountain View would not win any prizes for interior decorations. Would not win a Michelin Star either. But the food was good, served quickly and they had an amazing choice of various rolls - my greatest marriage betrayal happened there when my wife ordered a Microsoft roll.

I’ve enjoyed dropping by for a quick lunch or evening bite when I was staying nearby. It was closed in 2019.

The Bay Area food scene can be amazing but from what I have heard is also fairly brutal - especially for places on the main streets where the rent is high. Businesses open and close all the time and we just need to accept that the places we cherish and remember fondly might not be there when we visit next month - so go to your favorite place and make memories.

Main plan

To refresh your memory; we are implementing a coverage guided fuzzer. In order to gather the coverage we need to track the execution of the binary - by doing this we obtain information which parts of the program were executed and which ones were skipped. I’m aware that this is a terribly simplistic explanation but I assume that you already know that. If, however, you would like to go over the coverage gathering once again you can always read this great article written by h0mbre.

There are several methods of gathering coverage and previously we’ve used ptrace to collect the program trace with basic block resolution.

Resolution in this case means what is the granularity of the trace we are collecting. It can be as sparse as function or syscall we’ve reached or as fine as individual instructions. Chosen resolution impacts both the ability to guide mutation as well as, indirectly, the performance of the fuzzer.

Our chosen method had only one good characteristic - it was relatively easy to implement. Everything else was rather bad - performance was atrocious, we only gathered information about visited basic blocks, completely disregarding the order in which those blocks were visited or how many times. Today we are going to rectify at least some of the mentioned weaknesses.

Plan is as follows - we are going to alter the compilation stage of the program and insert small snippets of code. Those snippets will record every visited edge of the control flow graph and share this information through the shared memory with the fuzzing engine. More observant of you will immediately realize that we are essentially re-implementing AFL (and we are only 10 years late) and you will be right.

Shared memory intro

Shared memory is an operating system feature where two or more processes can have access to the same segment of memory. This allows the copy-less exchange of data and is exactly what we need in order to make our fuzzer fast.

Linux implements two interfaces for shared memory access - System V and POSIX. There are some differences between the two but, at least for our case they don’t matter that much. I’ve decided to use the POSIX variant because it’s newer and so there is no real compelling reason to stay with System V anymore

Fun fact: AFL uses System V. No idea why.

The basic routine when it comes to shared memory is that in one process you open the segment using smh_open(). Aforementioned function returns a file description. You can mmap() said descriptor as a readable/writable memory, so the program can make some use of it. The other process does exactly the same and as long as they agree on the segment name and certain flags they should be able to see the same memory fragment. For now we are going to skip over more advanced topics like mutexes, semaphores and queues.

We will begin by implementing two small C programs that will communicate with each other using this interface. First program called setter is presented below.

#include <fcntl.h>
#include <stdio.h>
#include <string.h>
#include <sys/mman.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <unistd.h>


#define STORAGE_ID "/SHM_TEST"
#define STORAGE_SIZE 32
#define DATA "Hello, World! From PID %d"

// SETTER
int main(int argc, char *argv[]) {
  char data[STORAGE_SIZE];
  
  sprintf(data, "Hello from %d pid", getpid());

  int fd = shm_open(STORAGE_ID, O_RDWR | O_CREAT, S_IRUSR | S_IWUSR);
  if (fd == -1) {
    perror("shm_open");
    return 10;
  }

  int res = ftruncate(fd, STORAGE_SIZE);
  if (res == -1) {
    perror("ftruncate");
    return 20;
  }

  void *addr = mmap(NULL, STORAGE_SIZE, PROT_WRITE, MAP_SHARED, fd, 0);
  if (addr == MAP_FAILED) {
    perror("mmap");
    return 30;
  }

  // Writing to shared memory
  size_t len = strlen(data) + 1;
  memcpy(addr, data, len);

  res = munmap(addr, STORAGE_SIZE);
  if (res == -1) {
    perror("munmap");
    return 40;
  }

  fd = shm_unlink(STORAGE_ID);
  if (fd == -1) {
    perror("shm_unlink");
    return 50;
  }

  return 0;
}

Going through the code we see that we’ve started with defining two constant values - the name of the shared segment as well as the length. It is important to keep that in sync between the setter and the getter otherwise we won’t be able to communicate. The size is also important - I don’t have to tell you what happens if you try to read or write to a memory that was not mapped correctly. Your fuzzer will produce its first crash, but probably not the one you would hope for.

As I’ve already mentioned - we need to open the shared segment first and we do that by using the shm_open(). Arguments are, in order, the name of the segment, flags and the mode. The name we’ve already mentioned but it’s worth remembering that those names can be seen as essentially file names. Using null bytes or slashes is generally discouraged. Flags control the way we are opening the segment - writable, readable or perhaps create it in case it does not exist. The mode only plays a role if we are creating the file and it sets proper permissions.

After successfully opening a share we can map it as a memory using the mmap(). Before we do that it is recommended that for newly created shares (O_CREAT flag) we call the ftruncate() to resize the share. Forgetting about this step and trying to read or write to the memory will leave you with a SIGBUS and interesting debugging adventure.

Speaking about the memory mapping - there are several flags controlling the behavior and the properties of such memory. It’s best to consult man or Michael Kerrisk book in order to get a full picture. This is especially important if you move between C and Rust because certain behaviors like MAP_ANONYMOUS might not exactly work as expected.

As for writing to said memory - mmap() returns a void pointer you can use freely. Well, almost freely as you need to remember about the size of the memory you’ve just mapped.

Last few lines are pretty easy to understand- being responsible programmers we do the cleanup by un-mapping the memory with munmap() and close the shared file using shm_unlink().

The getter code will be roughly the same except the memory reading part. Implementing it is left as an exercise to the reader.

Debugging tip: in the Linux system you can visit the /dev/shm directory where you will find all active shared memory segments.

Now, if you run getter and setter at the same time (you can help yourself by strategically inserting sleep() into the setter) you will notice we have exchanged the string using the shared memory interface.

Fuzzer part

Knowing how the shared memory works we can start implementing our fuzzing engine. I didn’t want to overly complicate the one I wrote in the previous parts, therefore I’ve decided to start a new one from scratch. This will also give me a chance to implement it more cleanly this time. Or actually, a bit later because right now we are going to sprinkle code with occasional unwrap() and expect() to make reading it a bit easier.

We begin with the part responsible for running the fuzzing target and gathering the coverage information stored in the shared memory. You can see the entire code below.

use std::env;
use std::ffi::c_void;
use std::process::{Command, Stdio};
use nix::fcntl::OFlag;
use nix::sys::mman;
use nix::sys::mman::{MapFlags, ProtFlags};
use nix::sys::stat::Mode;
use nix::sys::wait::waitpid;
use nix::unistd::{ftruncate, Pid};
use core::num::NonZeroUsize;

const MAP_NAME: &str = "/fuzz.map";
const STORAGE_SIZE: i64 = 64 * 1024;

fn main() {
    let runtime = env::args().nth(1);

    // open shared memory
    let shm_open_flags = OFlag::O_CREAT | OFlag::O_RDWR;
    let shm_open_mode = Mode::S_IRUSR | Mode::S_IWUSR;
    let mem = mman::shm_open(MAP_NAME, shm_open_flags, shm_open_mode)
        .expect("Failed to open shared memory");

    // resize the file to L1 cache size
    ftruncate(&mem, STORAGE_SIZE).expect("Unable to resize file");

    // map the shared memory as a memory region
    let mmap_prot = ProtFlags::PROT_READ | ProtFlags::PROT_WRITE;
    let mmap_flags = MapFlags::MAP_SHARED;
    let var = unsafe {
        mman::mmap(
            None,
            NonZeroUsize::new(STORAGE_SIZE as usize).unwrap(),
            mmap_prot,
            mmap_flags,
            Some(&mem),
            0,
        )
        .unwrap()
    } as *const u8;

    if let Some(r) = runtime {
        println!("Running fuzz target: {}", r);
        let p = Command::new(r)
	        .arg("ABC")
            .stdout(Stdio::null())
            .stderr(Stdio::null())
            .spawn()
            .expect("[!] Failed to run process:");
        let pid = Pid::from_raw(p.id() as i32);

        match waitpid(pid, None) {
            Ok(status) => {
                println!("[{}] Got status: {:?}", pid, status);
                println!("Reading from the shared memory...");
                let trace = unsafe { 
	                std::slice::from_raw_parts(var, STORAGE_SIZE as usize) 
	            };
                for x in 0..128 {
                    if x != 0 && x % 32 == 0 {
                        println!();
                    }
                    print!("{:02x} ", trace[x]);
                }
                println!();
            }
            Err(e) => {
                eprintln!("Error waiting for pid: {:?}", e);
            }
        };
    }

    unsafe { mman::munmap(var as *mut c_void, STORAGE_SIZE as usize).unwrap() };
    mman::shm_unlink(MAP_NAME).unwrap();
}

For starters, we are going to use the nix crate to easily access several system functions that we are going to need. There are several elements that you should already be familiar with, like the usage of mman::shm_open(), ftruncate() and mman:mmap() to, respectively, open the memory share, resize it to defined size and map it as a variable. Figuring out the flags and modes also shouldn’t take you too much time. Just like in our programs written in C, cleanup operations are handled by mman::munmap() and mman::shm_unlink().

First thing that requires explanation is the number of unsafe annotations we were forced to use. This, however, is hardly a surprise - after all we are essentially operating on a raw pointer that our program knows nothing about. As you can imagine such pointers don’t translate well into the Rust world so we need to convert it to something that the rest of the program will be able to use. We do this by calling std::slice::from_raw_parts() and supplying the starting address and the size in bytes. Thanks to this nice function we actually end up with a slice of u8 values we can freely read from. Of course - this comes with a huge risk hidden under yet another unsafe annotation. One thing you might be wondering is - but what about the type? If we look back at how we have mapped the memory you will see that we cast the resulting value into a u8 pointer. Rust is smart enough to infer that the slice we will be operating on contains values of this type.

Rest of the fuzzing engine, for now, is not very interesting - we basically just run the provided target binary and wait for it to finish so we can read the shared memory. Now, if you adjust the C program that we have written previously so the shared memory name and size matches things should work together. Running the fuzzer with the setter as an argument should result in printing out a hex representation of the shared memory modified by the child process. Just like on this image

Instrumentation

Now we are reaching the hardest part - how to convince the compiler to insert a set of instructions of our choosing into the binary composed of the source we have very little intention of modifying. What would be previously a 12-part series of compiler internals (that would probably be way over my head and cost me sanity), thanks to the great people responsible for clang and llvm, will be just a few lines of code and one weird makefile. Turns out that clang already comes with the interface for writing code instrumentation. Lets see how this works in practice by reading this simple code:

#include <stdio.h>
#include <stdint.h>
#include <sanitizer/coverage_interface.h>

extern void __sanitizer_cov_trace_pc_guard_init(uint32_t *start,
                                                    uint32_t *stop) {
  static uint64_t N;  // Counter for the guards.
  if (start == stop || *start) return;  // Initialize only once.
  printf("INIT: %p %p\n", start, stop);
  for (uint32_t *x = start; x < stop; x++)
    *x = ++N;  // Guards should start from 1.
}

extern void __sanitizer_cov_trace_pc_guard(uint32_t *guard) {
  if (!*guard) return;  // Duplicate the guard check.
  printf("Edge: %p %x\n", guard, *guard);
}

We see that there are two major functions that we define: __sanitizer_cov_trace_pc_guard_init() and __sanitizer_cov_trace_pc_guard(). Let’s start with the latter one - this function will be inserted by the compiler into every edge in the control flow and the *guard will point to a unique memory location - different for every edge. If you are wondering what can we find if we follow this pointer it’s time we look into the other function. It gets inserted by the compiler as a module constructor into every DSO. The start and stop parameters mark the area where all the guards for the entire binary are located and we can set those guards to whatever values we want. In our case I’ve simply gone for the incremental values.

Now, in our case we went for a fairly simple approach where we instrument only took a branch, but if you look at the documentation there are multiple other options - you can instrument other operations like comparison, store or dereferencing a pointer. Don’t let the Experimental labels discourage you and, well, experiment. I sense that some interesting ways to track coverage might emerge from this approach.

Knowing what we want to write it’s time we look at how to combine it with some other program. Admittedly, I haven’t had a time yet to use it against some real project as I was mostly working with samples I wrote on my own. Still, the same principles will apply and probably modifying at least one Makefile is unavoidable. In the meantime let us look at the one I wrote for the purpose of this article.

CC=clang
CFLAGS=-Wall -lrt

CFLAGS_INSTR=-fsanitize-coverage=trace-pc-guard,no-prune

## Universal rule for all cases
case_%: sample_%.o instr_%.o
	$(CC) $^ -o $@ 

sample_%.o: sample_%.c
	$(CC) $^ $(CFLAGS_INSTR) -c 

instr_%.o: instr_%.c
	$(CC) $^ -o $@ -c 
	
## Cleanup
.PHONY: clean

clean:
	rm -rf *.o 

I’m well aware that this one would not win any awards. Still, it does its job and I was semi-proud to make it work for all the samples and instrumentation variants I was writing without adding extra targets. Anyhow, we should start the analysis by looking at the instr_%.o target. For the instr_1.o file the compilation will resolve into clang instr_1.c -o instr_1.o -c. For those unfamiliar with the -c argument - it will compile the code into an object but without a final linking stage. We do the same to our sample code (in this case stored in sample_1.c) but in this case we provide a few additional flags like -fsanitize-coverage=trace-pc-guard,no-prune. This instructs the compiler to insert appropriate instrumentation where necessary. In the last stage we link both files together producing the final binary - case_1.

You might be wondering why I have not provided sample code that we add the instrumentation to. I believe that everybody should try to instrument their code of choosing. In my case I have a sample with a series of nested ifs that look for a certain word passed as a program argument.

One thing worth explaining is the no-prune option. Looking at the binary compiled without it might surprise you a little bit when some of the edges will be left without instrumentation. This is just an effect of the compiler trying to reason about redundant entries. In general I compile my code without any instrumentation pruning just to be sure everything is covered, but this is a fairly interesting topic that deserves at least a short note on its own.

Putting it all together

We’ve reached the phase where we adjust the instrumentation code to work with our fuzzer. We can see how it is done in the code below.

#include <fcntl.h>
#include <stddef.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/mman.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <unistd.h>

#include <sanitizer/coverage_interface.h>

#define STORAGE_ID "/fuzz.map"
#define STORAGE_SIZE 64 * 1024

void *addr = NULL;

void unmap() {
  munmap(addr, STORAGE_SIZE);
} 

extern void __sanitizer_cov_trace_pc_guard_init(uint32_t *start,
                                                    uint32_t *stop) {
  // Setup guards
  static uint64_t N;  // Counter for the guards.
  if (start == stop || *start) return;  // Initialize only once.
  for (uint32_t *x = start; x < stop; x++)
    *x = ++N;  // Guards should start from 1.


  // Setup shared memory
  int fd = shm_open(STORAGE_ID, O_RDWR, S_IRUSR | S_IWUSR);
  if (fd == -1) {
    perror("Failed to open shm share");
    return; 
  }
  
  addr = mmap(NULL, STORAGE_SIZE, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
  if (addr == MAP_FAILED) {
    perror("Failed to mmap file");
    return;
  }
  
  atexit(unmap);
}

extern void __sanitizer_cov_trace_pc_guard(uint32_t *guard) {
  if ((size_t *)addr && *guard) {
    // place a bit in a map
    uint8_t *map_ptr = ((uint8_t *)addr + *guard);
    *map_ptr  += 1;
    printf("writing to: %p\n", map_ptr);
  }
}

There are several elements that should be already known to you like initializing guards and obtaining a shared memory segment. The only addition here is the function responsible for un-mapping the memory when the program exits. Because clang instrumentation does not offer a destruction function we have to register one on our own using atexit().

The edge guard function is a bit more interesting than the previous one but still fairly primitive. Remembering that we have initialized each guard value with incremental number we can basically treat our shared memory segment as a simple bitmap and mark each edge as a single byte (not bit, because we are also counting number of occurrences). This approach will spectacularly fail in the program with more than 64k branches, but I think we are still far away from that point.

Running the instrumented code under the fuzzer produces the following output, clearly demonstrating that we have managed to successfully implement native code instrumentation.

Summary and future plans

We have a working mechanism to track coverage and report the results via a shared memory. There are few things that need improvement but we are already in a pretty good place. First of all, we are already tracking coverage on the branch level so we don’t have to do some weird bit shifting on basic block id to get this information. Second, with branch coverage pruning mechanism on by default we are mostly tracking the branches that matter so the 64k branch limit is far from being a blocker. Still, in the next parts I would like to prevent our instrumentation from crashing if there are more branches. In addition, with the current statically encoded shared segment name we can only have one fuzzer and one target running and I would like to amend that in the future version.

Besides that, in the next part you should expect all the other elements of the fuzzer coming together. If time and space permits I would like to focus on performance measuring and perhaps even profiling.

I was never a pro Vim user - I think I’ve started using plugins maybe two years ago and I’ve never managed to successfully configure a code completion feature. Few months ago somebody showed me the vim successor - helix. It quickly became my go-to editor for situations where breaking VS Code looks like an overkill.

I don’t write nearly as much code as I read so code navigation is a primary feature that I look for. I was happy to find that helix in this department has a lot to offer.

Symbols

I typically start the code review by looking at functions and structures if something catches my eye.

To display symbols in a given workspace just press space followed by a s or S - depending if you want to limit yourself to a currently open file or you want to operate in the entire workspace.

Find definition/declaration

Another action that I perform quite frequently is finding a definition/declaration of the function that I’ve encountered while reading code.

A little bit about the semantic used in this article - whenever I tell you to press xy it means to press letter x followed by the letter y - they don’t have to be pressed simultaneously, just follow each other.

To do that get the cursor on a code symbol and press gd or gD - depending if you want to get declaration or definition. You might want to ask what is the difference - in all the languages except C/C++ there is none.

Show references

Once you have the function reviewed you probably want to check where else it is being used. To find out get the cursor on the function name and press gr. Get on the function name and press gr

Jump/display structure

Displaying structure works the same way as displaying definition of a function. Move your cursor one the structure name and press gd.

Unfortunately, right now it’s not possible to display popup window with the entire type definition alongside the main coding view - unless of course you are prepared to use two different windows at the same time.

Go back

Now, the main problem with navigation in various editors is not moving forward - it is moving back. I happened to me multiple times that while chasing some parameter I’ve found myself six levels deep and not sure how exactly to go back. Over the years I have tried multiple different methods and bookmarks plugin in VS Code was on the top of my list. That is until I discovered jumplist. Every time you execute a command that navigates to a different place in the code your latest position gets saved into a jumplist.

You can display jumplist in quite a simple way - just press space followed by j - that will show you all the jump points. The most recent one is always at the bottom. You can also add your own jump point by pressing ctrl+s. If you want to speed up your workflow you can also press ctrl+o to jump back just one step.

Summary

There was important thing that needs to be said - most of those features depend on the presence of Language Server - run hx --health to check if your configuration support given language.

I had no problem navigating code in Rust and Go but C/C++ had me curse clangd multiple times - that is until I’ve discovered the reason. It only works on self-container files. Read this issue if you want to understand the problem better and this article if you are looking for a solution proposal.

The company I’ve joined was very different from the one we are seeing today. Or maybe it’s just that my perception has changed while the company stayed exactly the same. After all, when I joined it was already a multinational corporation with more than 30 thousand employees. While it thought about itself as a startup it was very far away from being one.

Don’t get me wrong - I never had any illusions what the goal of the company was. Plain and simple - to earn money. But you can earn money and you can earn money. Google I’ve joined felt like the former and somewhere it changed. Or we have.

Still, I want to tell a story about how I’ve even ended up there, what I have been doing through those years and what I think about the whole experience. 

This is a personal story told from the point of view of a single, unimportant engineer. If you are expecting some shocking revelations most likely you won’t find it here. I’ve also decided not to mention anyone by name. It is not because I’m going to say something bad about them -  quite the contrary. I just don’t know if they want to be part of the story or to be mentioned. Instead of reaching out to each individual I’ve just decided to make them anonymous. And the last caveat - human memory is imperfect and every event can be seen and remembered differently. Please forgive me if some of the accounts are inaccurate or you see them differently.

Getting into Google

In 2012 I moved to London to start a new job. While long term planning was never my strong suite my intention was to stay there for at least a couple of years. This was the first time I’ve worked abroad and while the initial time was very difficult everything eventually smoothed out and I’ve started enjoying my new environment. In October, when I felt I had a certain rhythm and stability I got a call from a recruiter that started a new adventure.

At that time google security team had a reputation of being the place where everybody wanted to work but very few people has managed to get into. I have never considered myself good enough to be noticed, not to mention invited to interview there. I felt both flattered and scared for failing miserably. Still, one does not refuse such an opportunity. After a brief chat I’ve promised to send my revised CV and wait for a phone screen to be scheduled.

Interesting fact was that the initial role I was offered only mentioned a US location. Fortunately I already knew some security people working in the Zurich office so I’ve managed to convince the recruiter to place me there. Moving to another continent felt a little bit too much at that point in life. Why am I even mentioning this? This will be important later on.

Speaking truthfully the closer to the phone screen I was the more uneasy I felt. I have heard many stories from the people who have tried and failed to clear it. To keep me on my toes google recruiters have postponed it twice. This delay and postponing will become a hallmark of the entire process. At that time it was mildly annoying but after seeing the process from the inside it became clear what was the reason behind it.

Overall, I don’t think the first phone screen went very well. I mean, it went well enough to be followed by a second round but I wasn’t very happy with myself. I specifically remember that I was struggling with some of the questions and basically had to ask the person interviewing me if I can talk about a similar topic instead. In this case instead of talking about establishing the SSL connection and key exchange I ask if I can maybe talk about SSH key exchange because the flow of the process is similar. This was a teachable moment and I’ve tried to follow this principle in my future career - allow the candidate to demonstrate technical acumen through the things he or she knows instead of the ones you have planned for them.

First phone screen led to the second one and I think that I did a bit better during that one. Still remember butchering some of the answers, especially struggling with some code samples with an obvious integer overflow. That was another teachable moment and now searching for arithmetic operations performed on variables being passed to malloc calls became my go-to strategy in bug hunting process. Years later when I actually asked about my performance on those phone screens and all I got was a very simple quip - “you must have done good otherwise you would not be sitting with us”. 

By the end of November I finally cleared phone screens and was invited for a round of on-site interviews. There was one small complication. The interview was about to happen in Mountain View, California but at that time I had no US visa. In a speedy fashion I’ve filed the paperwork and managed to get an embassy appointment. Within 2 weeks my visa was approved. Because of the upcoming Christmas holidays I’ve arranged the travel early January. Google was kind enough to arrange plane tickets, hotel for 3 days and even offered to get me a car. Not having an experience of living in California I’ve refused the last part of the offer. I think this was the first and the last time I’ve ever used a Taxi in the Bay Area and it made me realize why Uber became so popular there.

My plane landed on Monday and I had a whole Tuesday to rest and prepare for Wednesday. I spent that day in the hotel room reading some books and various other materials to prepare. I don’t recall exactly what I was reading but I’m pretty sure there was “The Tangled Web” and I’ve brushed up on all the topics I didn’t answer very well during my previous stages. Oh, algorithms and data structures as well. The reason for that was a dinner I had in December. For Christmas break I traveled to my hometown and met one of my friends who already worked for Google. Upon his advice I’ve spent a considerable amount of time learning about various algorithms. Given my lack of formal CS education it was a bit harder than it sounds. All that work was for nothing as I haven’t got a single coding question.

On the big day I got a total of four different interviews and lunch in between. The interviewers were covering a wide range of topics - starting from network security, through designing a single sign-on system finishing with a sizable portion about web security and some low level exploitation to spice things up.

Funniest part happened during the first interview. I need to mention that when I was invited to an on-site interview I kindly asked if it would be possible to have it done in Zurich. After all Switzerland was just a 2 hour flight away and did not require a visa. I was however told that there are not enough security engineers there to interview me. Imagine my surprise when, in California, I was led to a conference room and informed that the first interview was going to happen over the teleconference system with an engineer from Zurich. Turned out to be an exception as the three remaining interviews were conducted in person.

When the taxi took me back to the San Francisco airport I was in a fairly good mood. I felt that this round went pretty well and I was eagerly awaiting the decision. Of course there is a possibility that I’ve barely slipped through thanks to dumb luck and my optimism was unfounded. We will never know.

Waiting for the final result was exhausting. Apparently the hiring committees were so swamped that they haven’t managed to process my case during two separate sessions. It took them roughly a month to get to it. Due to the fact that my recruiter was on holiday that week, news was delivered to me by another one. She was very nice, walked me through the offer and explained the whole process that was about to take place. All in all it was a good offer but initially I wasn’t swept off my feet. I think it speaks more to the difficulty comparing offers between companies and countries than to google generosity. Factors like the tax system, costs of living and differences between benefits can sway your calculations wildly in all directions. Also, at that time knowledge about levels, bonuses and RSUs wasn’t so widespread among the general population so I wasn’t really understanding all the components. In the end, still not believing my luck I’ve accepted the offer without any negotiations.

Next three months were fairly busy - I had to bid farewell to my current employer and my colleagues, close all the matters in the UK and prepare to move to yet another country. Good part about that was the relocation package that Google has provided. On my last day in London a team of three people showed up, packed all my belongings to a container ready to be shipped to Zurich. My wife and I, with two suitcases, went to the airport to follow the same route. Our temporary apartment was already waiting for us.

My starting day was set for Monday but I’ve managed to get in touch with some of my friends earlier and set up lunch for Friday. Somewhere over coffee I asked about one thing that wasn’t very clear for me - when do I sign the employment contract? That prompted a series of questions and events that made the entire HR department scurry.

What most likely has happened was miscommunication between two recruiters. My lead recruiter must have assumed that all the paperwork was sent by the recruiter who called with the offer. Sadly, the other recruiter must have assumed otherwise and in the end I got no paperwork to sign. Good thing that the rest of the machinery concerning relocation was operating well.

I remember thinking that this is a great start - I was jobless and homeless (minus the temp apartment) in the most expensive city in Europe. Mind you, in the city the language I did not speak. In the end everything ended well - the HR department managed to prepare the contract and all the paperwork in record time and everything was signed before Monday.

Early next week I’ve showed up on the reception, had a picture taken, was given a temporary access badge and began my adventure as a freshly minted Noogler.

Continuation - including the teams and projects I’ve worked on during my time will be covered in the second part. It’s not very clear when I will have time to write it but I hope it takes me less than it took me to write this part.

Imagine a system or a process with some kind of property. This might be a service for buying books or a process for provisioning access to something. At some point someone has decided that this system must have 99.999% availability or maybe it was decided that every ticket in that queue will be addressed within 24 hours. For a period of time this property is upheld and everything works great. Inevitably however the system starts failing. There might be multiple reasons for that - maybe the complexity increased beyond the initial assumptions. Or maybe we are getting three times more tickets in a given queue than we’ve used to. It doesn’t really matter but the effect is easy to predict - our metrics start to go below the desired level.

Suddenly, a hero appears. It might be an individual who has decided to uphold the property no matter what. Such a person starts working extra hours to meet the goal, sets up extra monitoring, rolls back flaky builds, prunes logs manually, cull the overcommitted thread pools or sits on pager duty all weekends.

Saving a day on a weekly basis becomes addictive. Our culture praises heroism - toil and sacrifice are seen as an ultimate path to success. Medias are full of people telling war stories about 14, 16 or even 18 hours long working days. Eating cold pizza for breakfast is somehow seen as something glorious. One thing that nobody mentions however are associated costs.

First of all - every act of heroism masks the systemic and structural deficiency. By treating the symptoms you are masking the root cause of the problem. Maybe the property you are so desperately trying to uphold is no longer relevant - just nobody told you that. Or maybe there are other causes and the system needs to be re-engineered. One thing is however sure - you are heading for much bigger issues later on. What happens with the system when you decide to move on might become a huge problem for the organization.

Another aspect is that heroism is bad for your mental and physical health. Working a 80 hour week is unsustainable and sooner or later the consequences will catch up with you. Burnout and stomach ulcers are not as fun as you might think. If you are in your twenties and believe you are immortal, think about it from a different angle. Nobody has ever been promoted for doing repetitive L3 type of jobs - regardless of how many hours you put in.

The solution to that problem - let it fail. Best case you will discover that what you were heroically trying to maintain is not that important after all. Worst case the system fails and the management realizes the need for changes - ones that do not require a personal sacrifice.