So far MOP3 was fine with a basic round-robin scheduler. If you don’t know, a round-robin scheduler is a scheduler that has a list of runnable processes and picks the next item from it to execute, but when the end is reached, the algorithm wraps around and starts from the first item.

Simple flow graph:

Round-robin is simple and easy to implement but quickly becomes insufficient as the operating system grows in features, subsystems and drivers, which increases the amount of concurrently running apps. By having no priorities, everything equates to the same priority. This means that your text editor, which you’re currently typing characters into is as important for the scheduler as a USB controller daemon/poller, which does actual work only for eg. when a new device is plugged in (which is very infrequent). By having priorities, we can tell the scheduler that this USB poller process can be run less frequently, than this other app and thus reallocate precious CPU time to more important tasks.

We’ve added these two new fields: the base priority number and the rewarding priority number. Our priority system will be reversed if you’re used to UNIX-style priorities, ie. our priorities can be taken at face value. UNIX priorities are: higher number = worse, lower number = better, which is quite counter-intuitive. We’ll be using priority numbers to denote actual priority in a logical sense (more = good, less = bad).

This is it! We’ve just implemented priorities!

Now that we have priorities up and running, there’s a still small issue. MOP3 has a mutex API, allowing processes to synchronize and enter critical sections exclusively, which is essential to writing good multithreaded code.

When one process owns a mutex, other processes that try to hold it too, fail and have to be put to sleep mode until they can receive the mutex. This is so that a process doesn’t spin mindlessly in place (there’s place and time for spinlocks though, but not here and not today).

Mutexes don’t play nice with priorities, because they can actually reverse them!

The core issue is that a higher priority task is blocked by a lower priority task.

This is quite complicated to explain via text, so I’ll leave this diagram here:

~Source: https://picknik.ai/real-time/priority%20inversion/roscon/2024/01/31/Real-Time_Programming_Priority_Inversion.html~

We initialize the IDE driver from the PCI layer, so we need to enable some stuff in order to get DMA support.

We need to enable bus mastering. By flipping bit 2. This allows the device to talk to our memory via provided addresses.

We also must get the port number, which we’ll be using to configure our DMA transfers. This info is located in BAR4+0 for primary IDE channel and BAR4+8 for secondary.

We of cource then have to pass bmbase and bm_support to our IDE driver init function and now we just have to modify the driver itself to work with that.

In conclusion, adding DMA support was fairly easy. I’ve put it off for a long time, because I was a bit scared to tackle it and didn’t understand the subject that well, but after having written the XHCI driver (which is all about DMA), I felt pretty confident!

After having tested the driver for a bit on real hardware, there is a definitive performance boost! As a benchmark I’m using sys:/sdutil -format-fat32 -d ide0, which now takes up to a minute on a 32GiB drive, where previously it was 2-3 minutes.

A big portion of a working kernel is the synchronization code.

In a fairly modern OS, you’d want to support SMP or Symmetric Multiprocessing (multiple cores), so that the user experience doesn’t feel clogged up. Also you can run more tasks concurrently.

Synchronization (or lack thereof ) is also a HUGE source of bugs, known as race conditions. What is a race condition? A race condition is, when two concurrent tasks try to work with the same resource, at the same time, for eg. one tasks modifies a variable, while another one reads it - then the reader may read a partially-correct value, leading to broken state.

So what do you do then?

You implement LOCKS.

There are many types of locks, but today we’ll focus on two types: big / and fine-grained.

What’s the difference?

The difference is scope / domain. Big locks don’t care about scope, they just restrict access to entire subsystems or even the entire kernel. An example of this would be NetBSD, which had a big project, aiming to "modernize" their networking code (ref: https://wiki.netbsd.org/projects/project/smp_networking/). This is a consequence of building on top of 4.4 (4.3?) BSD kernel, which were built in times, where SMP hardware was quite unpopular, so it took a lot of effort to rewrite the system in an SMP-friendly manner. All BSD systems have suffered from this and even pre-2.36 Linux.

All of these systems have switched to fine-grained locking, which on paper is better, because you have many tiny locks, which span over little critical sections. This allows the kernel to not block other CPUs/tasks for too long, which makes the system faster / able to handle more throughput.

The main idea here is that big locks are considered "bad". And yes, they are in a sense that they make critical sections system-wide, but on the other hand, they make concurrent code easier to reason about (there’s nothing to reason about) and maintain. Of course, this comes with a performance penalty, but check this out:

Nobody cares

Or at least in scenarios, like mine.

Fine-grained locking, while giving a big performance boost, is hard (really hard) and even pro-sigma-h4x0r-67 Linux kernel devs can’t get it right at times. When you have so many locks, it’s easy to loose your mind and forget things.

A big lock is not an issue for MOP3, because

Now we need to answer the question: when do we hold the big lock?

The syscalls look like this now:

We lock before entering a syscall and unlock on exit. Don’t worry, cpu_request_sched will unlock before it loses context due to switching. We also must enable interrupts during syscalls, so that device drivers can work properly. Other than that, not much has changed.

Now here things get a little more interesting. We lock as before, but we do so only in the case of an interrupt incoming, while we previously were in user-space. This is because when in kernel-space we’re already under the protection of the Big Lock. Locking in such circumstance would lead to a deadlock, because we would retake the same lock and block ourselves.

In conclusion, MOP3 will from now on use the Big Lock, because it’s easier to maintain as a solo developer and the penalty doesn’t affect the system, because of it’s premises - being a hobby OS and lacking sophisticated software stacks, which would require better performance/control.

Hope you’ve learned something useful about big locks vs. fine-grained locks!

Funnily enough, after some testing and playing around with the system, I’ve found that the big lock actually has improved performance significantly?

Why?

I think it’s due to previously having a lot of spin locks, which equate to a huge amount of atomic reads/writes. Such operations on x86 may cause a performance problem, because they have to be synced across all cores. Having a big lock once on entry and big unlock on exit reduces the amount of these operations. That’s just speculation, I don’t have any real numbers to back this up; it’s all based on anecdotal feel of the system, while using it and running test apps.

sdutil or "storage device utility" is an app to manage filesystems and partitions on a device. Right now it supports DOS partitions/MBR, but I plan to add GPT in the future too!

I always make sure everything works by testing my code on a real machine. QEMU is nice for quick, iterative development, but you have to run you code in a not so nice and sterile environment. That’s how some bugs come up, which you’d never encounter in an emulator.

To test sdutil I’d run the following sequence of commands:

After step 2, I could NOT BOOT into my machine! WHAT???

Yeah…​

I could not boot, because there was no boot menu. It was simply gone! Here’s a photo (excuse the bad quality):

_??????

So after some digging in my code and debugging, I’ve figured it out…​

The BIOS scans available devices, so that it can populate the boot menu with various options to boot from. It will see a SATA drive and so it will try to get it’s first sector and parse it as a Master Boot Record to find other bootable partitions. This is why GPT still has a legacy MBR btw. The issue here was that the American Megatrends' BIOS would try to parse my faulty MBR and instead of failing and just skipping the device, it would just hang before rendering the boot menu. WHY?? I don’t know, go ask AT.

So what’s the solution? My machine is bricked!

Well…​ Not quite. It is bricked as in the BIOS software breaks when parsing the MBR, but we can unbrick the PC if we manage to wipe the MBR out.

I had to carefuly remove the 32GB M.2 drive. Here’s what it looks like:

It’s well past february of 2026, so we’re fine ;).

Then I had to go to a local electronics shop and buy an M.2 SATA → USB adapter. The plan is to simply plug the M.2 drive like an USB stick into my dev machine and use dd to wipe it out.

130 PLN/ZŁ later I have this:

Now we can clear the drive with dd. It takes like 10 minutes to do so.

And now WE CAN FINALLY HAVE THE BOOT MENU!! LET’S GOOOO

When writing sdutil I’ve made a big and bold assumption that a modern BIOS would not care about CHS (Cylinder-Head-Sector) related fields of partition table entries (PTEs). I was only initializing LBA-related fields and it worked fine on QEMU, which uses SeaBIOS.

What I did to fix this, was basically I’ve set up the CHS fields to mean the same as LBA. Here’s a simple conversion function in C:

And now we just have to pass the CHS fields into this function. Now everything works! YAY!

So what’s the conclusion for to day?

Of course we’re talking about x86/PC machines here

The right hardware will be either very easy or really difficult to buy depending on what platform you’re developing for.

If your OS is 32 bit (ie. runs on i386-based machines) then you’ve got a problem. 32 bit machines are hard to get these days and are quite expensive due to their high collector value and vintage/retro freaks willing to overpay for this electronic scrap. I myself own 2 32 bit intel machines - Pentium S and the OG i386DX, but they barely work and I doubt they could handle constantly smashing the reset button and rebooting.

64 bit machines are more modern (duh!) and thus you get a lot of quality-of-life features like USB booting, which is miles better than writing floppies or CD disks, just to find out your kernel doesn’t work and discard them, producing a mound of plastic garbage in the process.

Just because a machine is 64 bit, it doesn’t make it better for development! A CPU is not everything, it is rather just one of many puzzle pieces, which make up the entirety of the PC. This is why you have to mind what peripherals your machine offers. A lot of modern PCs don’t have legacy PS/2 ports or a serial port - and you can forget about LPT (rip). Everything is USB these days, which means that to have some simple debug printing (or just any output for that matter) you will need a graphics driver or a USB driver to send data. As you can imagine, this is really difficult to have when your OS is taking it’s baby steps, while a serial PC driver is like 30 lines of code. That’s why it’s important to have these simple, primitive, yet super useful peripherals available.

We’re going to implement the bare working minimum of the ABI, just enough to make thread keyword work in Clang and GCC. The spec is more complicated than that. We’re going to implement static TLS (there’s also dynamic TLS, you can look up tls_get_addr if you’re interested in going further).

Also I’d like to share this article as a very useful resource regarding the TLS: https://maskray.me/blog/2021-02-14-all-about-thread-local-storage. It’s more generally about TLS, but made for a great learning resource for me and I really recommend you read it too.

Other resources:

Thread local storage is a type of storage in a multitasked application, where each task has it’s own copy of it, distinct from other tasks.

Although the application is accessing and modifying a global variable, it’s actually different memories being used under the hood. Each thread has it’s own copy to work with.

What is thread_local? In the pre-C23 world it’s a macro, which expands to the _Thread_local keyword, which is the same as compiler specific __thread in GCC and Clang.

We’re going to learn how the TLS works via reverse engineering. We need to understand it, before getting to Implementing it ourselves. Let’s look at the disassembly first, generated by Clang 21.1.0 on https://godbolt.org.

I’ve added some comments here, so everything is nice and easy to read.

What is fs:[0] (also written commonly as %fs:0 in GNU syntax)?

We’re going to refer to fs as %fs (GNU syntax), because that’s how I write my assembly, but you can look up the analogous syntax for you assembler (like nasm or fasm).

%fs is an x86 segment register. There are also other segment registers:

MSR mean Model-Specific Register. Intel basically wanted to add unstable features and didn’t want to clutter up their architecture with experimental slop. Some of the MSRs were useful enough that they made it into future Intel CPUs and stayed with us. Generaly speaking, MSRs control OS-related stuff about the CPU.

MSRs are used with the rdmsr/wrmsr instructions. The scheme is like so:

I’ve mentioned previously that the %fs and %gs registers can be written to by writing to an MSR - but which one?

The MSR we care about is called (in the Intel manual) IA32_FS_BASE. To address the confusion early on I’ll say that some people call it slightly differently, for eg. in the Xen hypervisor code it’s called MSR_FS_BASE. My kernel takes the definition header from Xen, so that’s why I will use Xen’s naming scheme, but IA32_FS_BASE would be the official name.

Looking at the file kernel/amd64/msr-index.h we can see a juicy #define:

The magic MSR number is 0xc0000100. Here’s how I’m using it:

The MSR helpers are written like so:

What we do is we swap out base value of %fs for each process and every process has it’s own TLS! When processes are switched, the new MSR_FS_BASE is written.

We’ve managed to establish what %fs is, but what %fs:0 is?

The authors of System V TLS ABI for x86_64 were quite smart. %fs CANNOT be accessed on it’s own, sort of. We can’t use it like a regular pointer to the TLS. We can only use segment registers with a displacement. So when we can’t use %fs, we can use %fs:0! %fs points to the TLS + 8 byte pointer back to itself, so then %fs:0 can become a pointer to the real TLS memory block.

Also, the TLS variable offsets are negative!

If this is too difficult to grasp (don’t worry, I’ve spent days banging by head against a wall mysekf), I’ll show you now the code, which handles the TLS in a bit. Now we’re going to take another detour to discuss how the TLS looks like from the perspective of the ELF file format.

I’m not going to go out of my way to explain the ELF format entirely - it’s out of scope for today, but I’ll link a useful article here: https://wiki.osdev.org/ELF. It’s a great read on the basics of the ELF format.

_{https://wiki.osdev.org/images/f/fe/Elfdiagram.png}

ELF has the so-called "sections". A section is a piece of data that makes up the final executable. A section can be .text where your executable code resides or .rodata where your read-only data sits (like string literals).

ELF also has a special TLS section. This may seem confusing, since why would ELF store some sort of TLS, when each task must have it’s own? The TLS section is actually a template/"meta" section. It’s not the actual TLS, but rather a template of how should the TLS be contructed.

For example:

The first printf will display 123, because the TLS template says that a shall have initial value of 123, but then the thread is free to modify it’s own version. It just starts out with what is provided by the ELF file.

Now let’s take a look inside the ELF loader:

And that’s it! we can use the TLS now in user apps!

This is the final result of our work!

Integrating fat_io_lib wasn’t so straightforward as I thought it would be. To understand the problem we need to first understand the current design of MOP2’s VFS.

I need it, because Limine (the bootloader) uses FAT16/32 (depending on what the user picks) to store the kernel image, resource files and the bootloader binary itself. It’d be nice to be able to view all of these files and manage them, to maybe in the future for eg. update the kernel image from the system itself (self-hosting, hello?).

Here are some screenshots ;).

MBus is a one-to-many messaging system. This means that there’s one sender/publisher and many readers/subscribers. Think of a YouTube channel - a person posts a video and their subscribers get a push notification that there’s content to consume.

This is the user-space API for MBus. The application can create a message bus for objmax messages of objsize. Message buses are indentified via a global string ID, for eg. the PS/2 keyboard driver uses ID "ps2kb".

ipc_mbusattch and ipc_mbusdttch are used for attaching/detattching a consumer to/from the message bus.

The trickiest part to figure out while implementing MBus was to implement cleaning up dangling/dead consumers. In the current model, a message bus doesn’t really know if a consumer has died without explicitly detattching itself from the bus. This is solved by going through each message bus and it’s corresponding consumers and deleting the ones that aren’t in the list of currently running processes. This operation is ran every cycle of the scheduler - you could say it’s a form of garbage collection. All of this is implemented inside ipc_mbustick:

As you can see it’s a quite heavy operation and thus not ideal - but still way ahead of what we had before. I guess the next step would be to figure out a way to optimize this further, although the system doesn’t seem to take a noticable hit in performance (maybe do some benchmarks in the future?).

In the UNIX shell there’s one very useful statement - set -x. set -x tells the shell to print out executed commands. It’s useful for script debugging or in general to know what the script does (or if it’s doing anything / not stuck). This is one thing that I hate about Windows - it shows up a stupid dotted spinner and doesn’t tell you what it’s doing and you start wondering. Is it stuck? Is it waiting for a drive/network/other I/O? Is it bugged? Can I turn of my PC? Will it break if I do? The user SHOULD NOT have these kinds of questions. That’s why I believe that set -x is very important.

I wanted to have something similar in TB, so I’ve added a setlogcmds function. It takes yes or no as an argument to enable/disable logging. It can be invoked like so:

Now the user will see printed statements, for eg. during the system start up:

In UNIX shell, it’s common to get the output of an application, store it into a variable and then pass that variable around to other apps. For eg:

In TB, I’ve decided to go with a stack, since I find it easier to implement than a variable hashmap. A stack can be implemented using any dynamic list BTW.

The stack in TB is manipulated via stackpush and stackpop functions. We can stackpush a string using stackpush <my string> and then stackpop it to remove it. We can also access the top of the stack via $stack. It’s a special magic alias, which expands to the string that is at the top. An example of stack usage would be:

As you can see, even though we’re on AMD64, we use int $0x80 to perform a syscall.

The technically correct and better way would be to implement support for syscall/sysret, but int $0x80 is just easier to get going and requires way less setup. Maybe in the future the ABI will move towards syscall/sysret.

int $0x80 is not ideal, because it’s a software interrupt and these come with a lot of interrupt overhead. Intel had tried to solve this before with sysenter/sysexit, but they’ve fallen out of fasion due to complexity.

For purposes of a silly hobby OS project, int $0x80 is completely fine. We don’t need to have world’s best performance (yet ;) ).

Now that we’ve gone overm how to write some (very) basic programs in assembly, let’s try to untangle, how we get into C code and understand some portions of ulib - the userspace programming library.

This code snippet should be understandable by now: ._start.S

Here _premain() is a C startup function that gets executed before running main(). _premain() is also responsible for quitting the application.

First, in order to load our C application without UB from the get go, we need to clear the BSS section of an ELF file (which MOP2 uses as it’s executable format). We use _bss_start and _bss_end symbols for that, which come from a linker script defined for user apps:

After that, we need to collect our application’s commandline arguments (like argc and argv in UNIX-derived systems). To do that we use a proc_argv() syscall, which fills out a preallocated memory buffer with. The main limitation of this approach is that the caller must ensure that enough space withing the buffer was allocated. 25 arguments is enough for pretty much all appliations on this system, but this is something that may be a little problematic in the future.

After we’ve exited from main(), we just gracefully exit the application.

MOP2 pipes are a lot like UNIX pipes - they’re a bidirectional stream of data, but there’s slight difference in the interface. Let’s take a look at what ulib defines:

In UNIX you have 2 processes working with a single pipe, but in MOP2, a pipe is exposed to the outside world and anyone can read and write to it, which explains why these calls require a PID to be provided (indicates the owner of the pipe).

ipc_pipewrite() is a little boring, so let’s not go over it. Creating, deleting and connecting pipes is where things get interesting.

A common issue, I’ve encountered, while programming in userspace for MOP2 is that I’d want to spawn some external application and collect it’s output, for eg. into an ulib StringBuffer or some other akin structure. The obvious thing to do would be to (since everything is polling-based) spawn an application, poll it’s state (not PROC_DEAD) and while polling, read it’s out pipe (0) and save it into a stringbuffer. The code to do this would look something like this:

Can you spot the BIG BUG? What if the application dies before we manage to read data from the pipe, taking the pipe down with itself? We’re then stuck in this weird state of having incomplete data and the app being reported as dead by proc_pollstate.

This can be easily solved by changing the lifetime of the pipe we’re working with. The parent process shall allocate a pipe, connect it to it’s child process and make it so that a child is writing into a pipe managed by it’s parent.

Now, since the parent is managing the pipe and it outlives the child, everything is safe.