JakUB - Jakub's Universal Blog

Rack wars aftermath

2011-12-23T22:08:00.002+01:00

In my older blog entry, I wrote about my struggle to buy myself a rack cabinet. After ditching the first recommended distributor for pretty much the same flaws as those of the rack manufacturer itself, I think I made some good business with Ceit.

Before:

After:

The Itanium 2 server is not yet mounted in the rack as I am still waiting for its rack rails. On the other hand, I got hold of a Sun Fire V100 in the meantime, which is now mounted in the topmost position. And if I behaved properly the whole year, I may even find a Sun Fire V240 under my Christmas tree, so the empty positions will not be empty for too long.

European bugaboo

2011-12-09T10:32:00.001+01:00

Our Eurosceptic fellow citizens love to argue against closer integration within the EU using the loss of sovereignty bugaboo. But is that something to be really afraid of? And is it really a loss as in 'I lost my wallet'?

The idea of losing sovereignty is a heavyweight weapon which is powerful enough to keep the common herd from even thinking of something greater that exceeds the current borders of the state they are living in. If some decision making is to be moved from the state-level to the European-level, it must be bad. Bad!

But what does state-level sovereignty mean to an individual citizen, how can he benefit from it? Does it make any difference if the decision is made on the state level or on the European level? In either case, the citizen does not have a direct control over the decision. In both cases, he can indirectly influence it in elections. Now the question is, which level is capable of making more competent decisions. The answer will differ depending on the subject. Decisions with potentially global impact will be better made on the global, i.e. European, level while decisions with only local effect are better to be made on local, perhaps even sub-state regional level.

As for the noun loss, I think it was carefully chosen to dress the bugaboo with duly scary rags. I think sharing or delegation of sovereignty are both more fitting words, because the member state is actively participating in the European-level decision making process and thus still has some say over the issues. On the European-level, it even has a say over the issues it previously didn't have any control of.

Now, is this something to be afraid of? Should the citizen be alarmed if the state's fiscal policy is regulated on the EU level? In theory, a whole will not cause self-inflicted wounds to its own parts, will it? Of course, this self-preservation feature needs to be achieved by a balanced power sharing.

For example, the Constitution of the United States was designed in such a way that one house of the US Congress honors the States populations, while the other gives each State the same rights. The US Congress thus follows two power sharing principles: the more populous the State, the bigger say, and all States are equal in some sense.

In Europe, we have a similar system. Member states are represented in the European Parliament proportionally to their population, while the Council of the European Union will have one minister per member state. From 2014, the voting system in the Council will change towards qualified majority voting, giving thus more power to states, while still considering their population sizes. The European system can be further improved to more resemble the US one, but it still contains the two above principles.

The US is a good example of successful integration. It has a balanced constitution which enabled nearly twice the number of EU member states to coexist together and effectively share sovereignty. Maybe the Americans succeeded because of good timing or because after Articles of Confederation, this was already their second attempt at forming USA and they were able to learn from their own mistakes. From the few Americans I know, I have never heard worries about e.g. Californians overrunning Wyomingites in decision making.

By the way, the European Union today shows most of the weaknesses of USA under the Articles of Confederation, doesn't it? For the European Union to work effectively, it is necessary to share more sovereignty and share it in a more effective way. For example, the qualified majority of US States can amend the US constitution. This way of going forward is unheard of in the Old continent. The EU Treaty can only be changed by ratification in all member states, which makes any enhancements painfully slow, if not impossible.

In closing, I wish for less ambitious and populist member state governments, greater leadership on the European-level and much greater sense of belonging to the same Europe.

Proliferation of racks of mass destruction

2011-10-14T11:19:00.000+02:00

At the moment, there are 5 items on my desk that should rather be placed in a rack cabinet. These include an Itanium 2 server, a couple of UltraSPARC workstations, a monitor and a pair of Sun 5 and 6 keyboards. I therefore started to actively search for a fitting rack. I figured out that I need at least a 32U, 600mm-x-900mm cabinet with a couple of shelves.

After some searching I found a Czech company which manufactures these cabinets. I noticed there were no prices on their web, only a decent PDF catalog and a configurator form for inquiries. After having filled in the form, I was expecting to receive a quotation.

To my surprise, this was not going to be the case. So far, I have received the total of 5 emails from them - none containing the desired piece of information. The conversation went like this:

Me: Hi, I'm interested in the rack cabinet model RMA-32-A69-BAX-A1.
First Triton sales: Please let us know what is the country of residence of the company which is making the inquiry.

(The inquiry form only asked for name and e-mail address.)

Me: I live in Beroun, the same country as you are in and I need the rack cabinet for personal purposes. What is the price?
First Triton sales: We don't have the black one in the stock, but can sell you the grey one which we have. If you are not a company, please contact our distributor using the following telephone number.

I did not feel like using a phone. I don't remember myself using a phone for the
sake of buying something computer-related for ages. Since they did not provide the URL of their distributor, I had to find it myself. It was not difficult at all, but to my further surprise, there were no prices on their web either.

Me: The color does not really matter to me. Can you just tell me the price?
First Triton sales: The black unit and the grey unit both have the same price. As an end customer, you need to contact the distributor to buy them.
Me: I found the distributor's web, but they don't list the prices either. At the moment, I just need the price. Is the price secret?
Second Triton sales: The price is not a secret, but it is a common practice for both us and the distributor not to list the prices on the web. I am sure they will tell you what the price is if you call them or write them an e-mail with your phone number.
Me: I probably underestimated the complexity of my inquiry as I was expecting a simple answer just as if I was looking for a price of e.g. a laptop. FYI, some of your competition are not that secretive about their rack cabinets.
Second Triton sales: Well, you see it. If you included your phone number in your e-mail signature as is appropriate, I would have already told you what was the price. I am not sure how I can further help you. You really should contact the distributor.

WTF? Turns out that I have been using my e-mail inappropriately for years! I am not sure how this is going to end. I suspect that Triton might be in a need for a brand new sales team, because the sales lady and the gentleman with whom I have spoken won't recognize a customer even if the custmer wore a label "I am a customer."

Next Generation Jermar, model B

2011-10-14T09:59:00.002+02:00

On October 2, 2011, I became a multi-father as my second son, Max, was born on that day shortly after 8am.

My first son, Vita, is a little bit jealous now because of his younger brother, but my and my wife's confidence that in the long term he will appreciate the new family member is high.

Debugging file system hang using kconsole

2011-09-10T23:37:00.001+02:00

I thought it may be useful to interested people if I demonstrated how the HelenOS kconsole ipc command can be used to root cause a deadlock in the file system code. Yesterday I had the opportunity to use this technique on ticket #373. In short, the mount command was hanging in an attempt to mount a MFS file system on a file-backed block device with the underlying image located on the root FAT file system.

Let's first have a look at the commands which lead to this deadlock:

With the deadlock in place, we can start debugging it by switching to kconsole and inspecting the IPC state of essential tasks. The first one to look at is VFS, which is usually task number 6:

The screenshot above reveals that VFS is processing two client calls and that it itself has made two calls to the FAT server. The first call with method 1032 is VFS_IN_MOUNT sent by bdsh as it was executing the mount built-in command. From the source code, we see that VFS needs to process VFS_IN_MOUNT by sending VFS_OUT_MOUNT to the file system with the mount-point node. This corresponds to the first of the two calls sent over phone 2. Let's now look at what the situation is at the FAT server:

We see three calls dispatched to the FAT server, the first one being the VFS_OUT_MOUNT. In reaction to VFS_OUT_MOUNT, FAT will send VFS_OUT_MOUNTED to the mountee file system. The mountee, in this case, is task 41 - the MFS server.

Since MFS is being mounted on a file_bd block device, MFS will want to do some reads from it before it answers the VFS_OUT_MOUNTED call.

This file_bd instance has its blocks in the image file located on the root FAT file system, so it makes VFS_IN_READ and IPC_M_READ_DATA calls to VFS in order to read the block. We already saw the former on the screenshot with VFS as the second dispatched call. We also saw the IPC_M_READ_DATA on the FAT situation screenshot (call with method 8) because it got forwarded to FAT.

In reaction to VFS_IN_READ, VFS will send VFS_OUT_READ to the file system with the respective image file, i.e. the FAT server. It is the call with method 1025 on the FAT situation screenshot above. It is this call which contributes to the calls column of phone 2 on the VFS screenshot by its second half.

Now the deadlock cycle is completed. As seen on the FAT screenshot, the connection fibril associated with the VFS phone 2 is processing the VFS_OUT_MOUNT call and cannot finish it because for that it also needs to start and complete processing of the VFS_OUT_READ call. On the other hand, it cannot even start processing VFS_OUT_READ, before it is done with VFS_OUT_MOUNT.

I have described a similar problem several years ago in this blog. The solution was the introduction of async sessions. So the million-dollar question is why this didn't work this time and why we see that the two requests which should be handled in parallel are sent over VFS phone 2 serially.

The possible answer could be hidden in the way how the first call is handled. And indeed, in vfs_mount_internal(), we find the following piece of code:

vfs_exchange_release(exch);
async_wait_for(msg, &rc);

The code is wrong because it ends the exchange before the answer to the VFS_OUT_MOUNT call arrives. That makes it possible for the underlying exchange phone to be reused for handling other requests, including those that need to be handled in parallel to the first one. With this realization, the fix is a piece of cake.

With Bobcat will come the features

2011-09-08T15:06:00.000+02:00

So far, we have resolved only about a quarter of all tickets planned for HelenOS 0.5.0, but judging only by that one quarter, the next release will be full of new and exciting features. This is mainly due to a quite favorable confluence of circumstances such as our participation in this years Google Summer of Code program or successful completion of several HelenOS projects at universities. We have not been sitting with our arms folded either.

Just a brief enumeration of my favorite features (a little bit more complete list can be found in the release notes that are currently in the making):

USB stack and drivers
new file systems (read-only Ext2, exFAT, ISO 9660, MINIX FS, LFN for FAT)
C toolchain (GNU assembler and linker + Portable C Compiler)
support for device removal

And having roughly three quarters of work - features and bug fixes combined - still ahead of us, gives us hope to see even more heavy-duty goodies:

RBAC based security scheme
support for NUMA
NIC framework and other networking improvements

My trace in HelenOS Camp 2011

2011-08-22T22:19:00.013+02:00

The coding camp is over and it's time to look back a little and describe the things that went into the mainline. For my part, the most interesting and also the most time consuming bits were related to fixing ticket #259.

The gist of the ticket was to change the way how open files are passed from one task to another, typically when spawning a new task and passing it its standard input, output and error files. Previously, this was achieved by exporting VFS-internal information - the VFS triplet composed of the file system handle, device handle and file system index - to the donor task. The donor task then sent this triplet via IPC to the acceptor task which in turn used it to open the respective files.

There were several problems with this approach, the most serious being the one that I didn't quite like it. But other than that, it was implemented as an afterthought in the absence of better alternatives. Reopening the file is not exactly the same thing as passing its open handle, because the file handle is associated with position, append on write flag and mode (not currently supported in HelenOS). If the donor closed its handle or did not even have the file open, the mechanism was susceptible to races.

In ticket #259 I called for a first-class VFS operation for passing device handles. The result of the operation should be that VFS duplicates a donor's file handle in the acceptor's table of open files and informs the acceptor about the new handle. Now, the real problem of course is how VFS learns about the intentions of its two clients and how it locates their data. We also want to prevent VFS from injecting a file handle against the will of any respective acceptor and, vice versa, importing a file handle against the will of any respective donor.

I started drafting the authorization protocol about a month before the camp during one of the rather frequent electricity blackouts. At that time I thought of introducing a specialized system IPC message for passing any sort of a handle in general between two clients of the same server. The message, something like IPC_M_PASS_HANDLE_AUTHORIZE, would be understood by the kernel (just like any other system IPC message). In this case, the kernel would be the trustworthy entity certifying that the donor task agreed with the acceptor task on passing a handle. I planned to store such a certificate, along with identification of both parties and the handle, in the donor's kernel address space, from where it would be later picked up by the donor sending another system message, for example IPC_M_PASS_HANDLE_REALIZE, to VFS. It was this second message which should make VFS perform the transfer. The only problematic question was how to organize such certifications within the donor. Should it be limited to a use-once right or rather should each task have a container for storing and managing certifications? Or should the kernel send the second message itself on behalf of the donor?

With the basic idea and some unanswered questions, I joined the camp (technically the camp started while I was already there sitting at my dining table).

I discussed my idea with Jiri and Martin to get some feedback. Jiri was traditionally skeptical and was frowning at me. Apart from having an alternative idea of turning HelenOS into a capability-based system (sigh), he also advised me to make the whole mechanism more generic. Instead of passing some sort of a handle, why not let the two sides agree on a protocol-specific change of externally kept state? I liked his idea so I moved from IPC_M_PASS_HANDLE_AUTHORIZE to IPC_M_STATE_CHANGE_AUTHORIZE and reserved the first three arguments of the message for the protocol-specific definition of the desired change of state. Martin was influential in solving the question of how the acceptor learns about the new state. I originally intended the donor would tell the acceptor, but there was a purely hypothetical risk of the donor intentionally misleading the acceptor by providing false information. Not so sure about why would the donor have bothered with such a mystification, but anyway, I came to realization that this final phase of the exchange is in fact an optional, protocol-specific synchronization between the acceptor and the server.

At this point (or slightly before that), I started coding. I first implemented the IPC_M_STATE_CHANGE_AUTHORIZE and its kernel and async framework support. The donor sends this message to the acceptor, which may be (and usually is) the child task. It packs the protocol-specific identification of the desired change of state into the message. Here comes one important detail: the donor also needs to identify the server process by presenting an open phone to it in the message. When sent, the kernel remembers the server phone and passes the message to the acceptor. The acceptor now has a chance to reject the offer or agree to it. If the acceptor agrees, it too needs to present an open phone to the same server in the answer. The kernel now checks that both the donor and the acceptor mean the same server task by looking at the two phones. If there is a match, it sends a per-task kernel event notification EVENT_TASK_STATE_CHANGE to the server task. Note that the use of a kernel notification here gets us rid of the problem with storing the certification in the donor task.

For this to be possible, I had to implement support for per-task kernel event notifications as HelenOS only supported global kernel event notifications. The difference is that in the case of a global kernel notification only one task can subscribe it (i.e. one task receiving e.g. the EVENT_KLOG notification) while it is technically possible for each task to subscribe to any of its own per-task events. In this regard, per-task kernel events resemble POSIX signals.

Back to the server task receiving the EVENT_TASK_STATE_CHANGE notification. The notification carries information necessary for the server to locate both the donor's and acceptor's client data and the description of the desired change of state. In connection with accessing some task's client data from the notification context, it was necessary to add functions for managing external references to the client connection tracking structures.

In case of the VFS server, it handles the EVENT_TASK_STATE_CHANGE notification by looking up the donor's and acceptor's client data which contain the respective tables of open files. It performs several sanity checks to verify that the file handle exists in the donor's name space and that there is a free file descriptor in the acceptor's name space. If everything looks good, it clones the vfs_file_t structure and points it to the underlying VFS node. Finally, it enqueues the new file handle in the acceptor's client data and broadcasts its condition variable to wake up any clients that may be waiting for it.

This brings us to the details of the acceptor-server synchronization. For this purpose, VFS implements a new interface VFS_IN_WAIT_HANDLE and libc makes it available to clients via the fd_wait() function. Clients invoking fd_wait() will simply block on their VFS client data condition variable until there appears the new file descriptor. VFS clients call fd_wait() after the reply to IPC_M_STATE_CHANGE_AUTHORIZE and before replying to the leading message such as LOADER_SET_FILES. Without a synchronization like this, the donor could exit before the server has a chance to duplicate its handle or it may be too early for the acceptor to query its new state.

I tried to illustrate the whole communication in a picture above. As a whole, it may seem more complicated than the previous solution that simply passed VFS triplets around, but is definitely cleaner. Moreover, HelenOS now features a universal mechanism for achieving a change of externally kept state between two clients of an arbitrary server.

Family background of synchronization primitives

2011-08-12T12:57:00.005+02:00

If you want to have some good Friday read about synchronization primitives and enjoy a couple of fitting analogies, make sure to read this LWN article. May be, you will want to learn more about Seqlocks, just as I did after reading the article.

Find 10 differences

2011-07-29T11:54:00.003+02:00

2011:

1970:

Milestone accomplished

2011-07-17T00:12:00.002+02:00

Yesterday my 22 month old son said the magic sentence Ano, tati (Yes, daddy) for the first time. This is a major progress from the previous series of ne (no), [ne]chci (I [don't] want) and combos like nééééééé, mamáááá, néééééé!

Losers assemble, real men compose

2011-07-15T15:44:00.006+02:00

There are basically two approaches to software development. Assembling the code by cherry-picking pre-made components or writing nearly everything from scratch. As usually, both have their cons and pros.

Assembling is suitable for undermanned, time-or-otherwise pressed projects or simply for projects with a rather narrow focus which is noticeable even on the base functionality level. By reusing third-party components, these projects avoid the Not-Invented-Here syndrome, which may be a good thing because it makes other people's work worthier and saves the project's own resources. On the other hand, it may be a bad thing too as the extensive use of non-homebrew base components destroys the identity of the accepting project and can introduce bloat in the form of various compatibility and glue layers, as well as the very much undesired burden of maintaining a large foreign codebase and its bugs.

For example, it turns out that our companion multiserver projects, MINIX 3 and GNU Hurd, already have or soon will have some parts assembled from:

old Linux networking stack (GNU Hurd)
old Linux device drivers (GNU Mach)
old Linux file system drivers (GNU Hurd)
L4's DDE framework for using contemporary Linux drivers (GNU Hurd and MINIX 3)
lwIP networking stack (MINIX 3)
NetBSD standard C library and base utilities (MINIX 3)

This bears questions like whether MINIX 3 without its traditional userland will still continue to be MINIX 3 or rather it becomes a NetBSD distro, and gives us some idea about why are some GNU Hurd's and GNU Mach's components stuck in times of Linux 2.2 or even older.

The other extreme is HelenOS, also an undermanned multiserver OS project and one in which I am actually involved. We tend to be affected by NIH and often find ourselves to be reinventing the wheel, but at the end of the day, here we go with our own implementations of the above components (networking stack, filesystems, drivers and device driver framework, and libc). Sure, they may not be perfect today or tomorrow, but they are native and unique to HelenOS and remain under our control and in our custody.

A European look at the incomplete GSoC student statistics

2011-05-27T11:53:00.004+02:00

There is an interesting graph on the Google Open Source blog showing the breakdown of this year's students by their country of origin. Even though the graph does not list countries with student participation below certain limit, it provides enough information to draw some interesting conclusions.

At the first sight, it looks like the US(191) and India(182) are miles ahead the other countries when it comes to the number of participating GSoC students. But when we take a second look and add the few European countries that do show up individually in the graph, namely Germany, Poland, Romania and France, we see that just these few EU countries combined are on par with the US. The final EU number will be probably much higher than that if we add students from the remaining 23 members (for instance, at least 2 for the Czech Republic).

Clearly, there is some great programming potential hiding in Europe. Now, if only Europe knew how to make use of it.

Hola, AVAST Software!

2011-04-18T16:59:00.006+02:00

Today was my first day in a new job. As the title suggests, I joined AVAST Software, the maker of the venerable antivirus program avast!. I will be working on a x86 virtualization technology related to this. Leaving the familiar grounds of C and Solaris seems a little dangerous at first. But hey, I will soon make my hands dirty from C++ and Windows kernel programming, which makes the risky business quite rewarding.

No more Solaris fixes or Hasta la vista, Oracle!

2011-04-12T11:25:00.010+02:00

I am not planning to make any more integrations to the Solaris source code repository in the near future, so I am glad that before my last day at work I managed to root cause and fix a couple of nasty race conditions in Solaris SPARC VM code. Too bad I may not share more details on this blog, but perhaps I can use some analogy. If we view the TSBs (Translation Storage Buffers) as apples, the Solaris kernel as the railways and linked lists as trains, my last integration can be described as fixing a problem in which the railways suddenly attached an empty wagon to a moving train without first stopping it and letting the locoman know, or in which the locoman was expecting a wagon full of apples to be attached, but the railways attached an empty one instead. Sure, it would be more nourishing to describe the above issues in the native terms, but it appears that for that we will need to wait some time.

Today was my last day in the office, even though technically, I will be an Oracle employee until Friday's midnight. Companies change and so do people. I am thus closing one chapter of my professional career. Stay tuned for an update regarding my new position.

We have been accepted!

2011-03-18T21:32:00.004+01:00

After three years of trying, HelenOS has been finally accepted as a mentoring organization for Google Summer of Code 2011. Hurray! Hurray! Hurray!

Fellow villain East-Europeans, mark your calendars!

2011-01-24T15:01:00.003+01:00

In addition to our favorite extra-curricular activity of pillaging and conducting forays of German-speaking border country, which we have been freely enjoying, unlimited by any state borders, since December 2007, we will soon celebrate another triumph. In May of this year, our hordes will completely take over the German and Austrian ditch-digging job market and make millions of German and Austrian workers jobless. Beware, the barbarians from the East are clustering together!

A decade of HelenOS

2011-01-02T13:16:00.006+01:00

In 2011, the oldest HelenOS code will turn 10. Those who were watching during its first decade, have seen HelenOS grow from a one-man adventure to a truly multi-team project involving enough people to fill a bus. HelenOS has gained substance and infrastructure in places where other similar projects have put only gilded fluff and left void. Instead of taking the easier path of imitation, it has mostly gone down the more challenging road of independent design.

Among others, the last decade has brought HelenOS:

a multiplatform kernel, a foundation on top of which everything else is built
an asynchronous communication layer, which interconnects everything
a basic user interface which allows the system to be used in a simple way
a file system layer so that we can import and export data to and from HelenOS
a modular networking stack, so that we can ping localhost and serve a web page
a device driver framework to unify work with devices and manage differences amongst various platforms in a systematic way
an interpreter of a high-level object oriented language with which one can extend the capabilities of the system beyond those it gained during compilation

Behind the items above and also the items that I failed to list, there are people who dedicated many hours, days, weeks and months of their time to making HelenOS better in one or another way. They were somehow able to appreciate the constant not-yet-finished quality of the system and could see the light at the end of the tunnel, which in turn got brighter and brighter. Without their help, HelenOS would remain just a one-man adventure it once was. I would like to thank all these creative and courageous people for what they have done so far.

Looking into the next decade, there is still a lot of work left to be done. We may have laid the foundations, but there is still a house to be built on them. Moreover, as our understanding of the overall system requirements changes and refines, at times it will become necessary to touch these foundations and make modifications to them, without tearing down the entire house.

At the beginning, I was suggesting that HelenOS has more substance and infrastructure where some other projects do not. But the opposite is true too. Also HelenOS has major functionality and, more importantly, usability gaps that need to be fixed.

For example, I would like to see the system become self-hosting and installable. A basic set of tools for administration will have to be created. There are ongoing projects that aim to introduce common concepts, such as security, that we have been carefully neglecting so far. As a true multiplatform project, we need to make sure that the features we develop are eventually available on all applicable platforms that HelenOS supports.

I am looking forward to the fruits of the research being done on HelenOS by others. This is something that I have always hoped to be doing myself, but unfortunately never had a good opportunity to do as the circumstances pushed me in a slightly different direction.

I'd be very glad if we managed to keep and perhaps even increase the interest among students and other enthusiasts and form an even stronger HelenOS community. Hopefully, the community will slowly change into a community of users and developers instead of merely a community of its own developers.

GCC has gone mad

2010-11-23T16:30:00.004+01:00

In my July post I praised the GCC ability to detect uninitialized variables, but now it seems I have to take this praise back. We discovered several cases of false negatives (i.e. missing warnings) about uninitialized variables in GCC 4.5.1 that look like regressions from the behavior of GCC 4.4.4 or perhaps were not detected even by the older version.

Make your own picture of the situation. The following is an example of one use of an uninitialized variable that goes undetected by the new GCC at the -O3 optimization level:

312 static int packet_reply(const packet_t packet)
313 {
314         ipc_callid_t callid;
315         size_t size;
316 
317         if (!packet_is_valid(packet))
318                 return EINVAL;
319 
320          if (size != packet->length)
321                  return ENOMEM;

And yes, packet is valid and line 320 is being reached. This and other examples can be found here:

Comparison with an uninitialized variable,
Passing an uninitialized variable to another function,
Function leaves its output argument uninitialized.

On the web I found the GCC meta-bug for these and related issues and it looks like this is currently a big problem in GCC. Perhaps I would get more warnings if I first built with -O. That's possible, maybe. But I don't want to run the compilation twice. I don't like arguments that say if we detected uninitialized variables more reliably, the compilation would get slower. I can wait a little longer if I get reliable warnings.

HelenOS gains momentum

2010-11-17T13:28:00.013+01:00

I have just gone through the inventory of all active HelenOS student projects and was impressed by the actual finding. Not only that there have never been so many students working on HelenOS at the same time, but the current number is quite high so that HelenOS as an academic project compares favorably with similar, but heavily funded, projects like MINIX 3.

	People	Projects
Semestral assignments	1	1
Software projects	9	2
Bachelor theses	1	1
Master theses	8	8
Doctoral theses	1	1
Total	20	13

The total number of students is actually 19 because one student works both on a team Software project and his master thesis and the above table counted him twice. Most projects figuring in the table have started only recently, which means that we are not accumulating unfinished projects. Quite the opposite: we observe an increasing trend in the amount of defended master theses:

Year	Theses
2006	1
2007	1
2008	2
2009	2
2010	3

I was wondering what is the reason for such a rise of interest in HelenOS among students and while there is probably no single correct answer, here are my suspects:

The project is in a stage in which it needs some larger frameworks to be developed; larger frameworks can be good fit for team projects of four or five students, such as the Software project subject at Charles University.
We had some interesting and high-quality project ideas up in our sleeves originally intended for the Google Summer of Code 2010, but since we were not accepted as a mentoring organization, these projects are now being picked up by regular university students, including students from foreign universities.
HelenOS is now a first-class project at D3S. The department's web site has recently undergone a major facelift and as a result of that, it mentions HelenOS in various places that are likely to be noticed by students looking for an interesting topic.
The whole HelenOS movement itself is like a black hole. With every successfully completed and integrated student project, it gets noticeably better and becomes more attractive to other students who in turn tend to start new projects.
Martin has likely changed his recruiting practices so that they appear more efficient now. Even though it remains unclear what change he made, it appears to work in synergy with the above.

The message is clear: if you are a student and your project or thesis topic is not related to HelenOS, than I am afraid you are not currently in :-)

The list of student projects and theses in progress is available on our wiki. If you actually happen to be looking for a topic, the same wiki contains various pointers to yet unassigned project ideas (here, here and here). Or simply just drop us a line!

Switching to a new version control sytem is not easy

2010-10-21T10:53:00.008+02:00

Today I read this LWN article about the pains of PostgreSQL switch from CVS to Git. It reminded me the lessons learned during a similar endeavor which happened about a year before the PostgreSQL move.

When HelenOS was about to switch to Bazaar in August 2009, its Subversion repository contained only some 4686 revisions, but our previous, rather unsuccessful, attempts to convert the repository to Mercurial made us start with a brand new repository rather than a converted one.

The fact is that we lost the continuity of our revision history, but on the other hand we did not end up with a twisted, possibly damaged, or even missing history. For our reference, we still keep the old Subversion repository around and we also have the Mercurial convert, which documents the steps needed to turn a medium-size repository of a centralized version control system with a handful of branches into a repository of a distributed version control system.

The conversion process using hg convert was very slow, tedious and error prone. hg convert is not to be blamed, however, because the problems stem from the fact that there is no apparent 1:1 mapping between Subversion and Mercurial repository entities. A couple of times, I had to stop in the middle of the conversion, create synthetic history (i.e. commits that never really happened) to emulate various repository events and then resume the conversion at the next revision. The events that were confusing the conversion software were:

changes of the repository namespace, for example when we switched from the kernel/trunk naming scheme to trunk/kernel,
starting a new branch, and
merging an existing branch into trunk.

In fact, the process was so tedious, that I decided to only fake history for the first type of events and threw the rest over the board. The result is that the Mercurial convert repository does not contain revision history for any but the main branch and it does contain some synthetic commits that mimic the various main branch renames and shuffling that happened in the past.

The convert is still useful though, because it at least serves as a backup for our old trunk history, but there is no doubt that some information was lost in translation.

The LWN article also touches another interesting point: it takes some serious mind-quake to stop thinking in the centralized repository way and accept the distributed nature of the new system. PostgreSQL, perhaps temporarily, decided to use the repository in a rather centralized manner and does not permit merges. The same is true for the OpenSolaris Mercurial repository, where the history is artificially kept linear. In our case, we wanted to go distributed from day one, but our initial clumsiness in distributed thinking lead to several merges that unfortunately swapped the left-hand side branch (i.e. mainline) with the right-hand side branch, back and forth. We learned from our mistakes though, documented the process and developed a Bazaar plug-in which prevents us or anyone else from repeating the same error again.

Erare humanum est, but there are also lessons to be learned from either the LWN article or, hopefully, this blog entry.

What do kernel engineers eat for lunch (when they work from home)

2010-09-07T12:31:00.006+02:00

Before (spaghetti, garlic, chili pepper, extra virgin olive oil, water, salt, parmesan, sweet basil):

After (Spaghetti aglio olio peperoncino alla Jacobo):

Can you guess what was the only non-authentic ingredient?

R.I.P. Sun Microsystems Czech s.r.o.

2010-08-31T17:09:00.003+02:00

Tonight, Sun Microsystems Czech s.r.o. will see its last sunset. Tomorrow, it will be already preying in the happy hunting grounds with Sun Microsystems Inc.

Mourning and funeral rites will take place at dusk at Legenda.

FAT on a diet

2010-08-18T21:51:00.006+02:00

Some time ago, we received a ticket that complained about large files taking too long to write to a FAT file system. Times of several minutes in a simulator were reported for writing a file which occupied a couple of megabytes. After some pondering about the problem, it became apparent that the problem is the FAT file system itself, more precisely the way it keeps track of blocks belonging to a file, and also the fact that HelenOS did not even try to do the smallest thing to alleviate the situation.

Used blocks of each file on a FAT file system are tracked in a linked list of consecutive clusters, where a cluster represents a fixed number of adjacent blocks. The use of the linked list structure in the on-disk file system format constitutes an interesting problem for the operating system, especially when doing sequential I/O. Whenever someone writes to or reads from some offset within the file, the operating system needs to find the block for that offset. Without any optimizations, this means walking the linked list from its beginning until the cluster and the block with the wanted offset is reached. In the worst case, one needs to walk the entire list until the last block number is found. Thus the algorithm for looking up the block with a single file offset in a file of n blocks has time complexity O(n). Now imagine that the application does not do this once, but searches the linked list repeatedly such as during sequential I/O like when loading an executable, creating an empty file, and copying or printing out an existing file. In these scenarios, the naive implementation will be traversing the list multiple times, starting from the beginning over and over again. Since each traversal is O(n), repeating it n times in the worst case gives us an O(n^2) algorithm where n is the number of blocks.

In comparison, for most disk file systems it is usually possible to devise a constant time algorithm for looking up a single block thanks to the on-disk index of used blocks. FAT as such, as I derived above, allows only for a linear algorithm because it only has the linked list.

The test case scenario in the ticket made me realize how unfortunate the situation was (I hereby admit that I was knowingly underestimating and derogating the issue prior to this experience, as Jiri can attest) so I resolved to fix it at least for the most common cases.

The fixes were actually two quite straightforward optimizations, both merged in changeset:mainline,624. The first optimization works by remembering the last cluster used by the file, making thus the append operation work in constant time. This comes handy when e.g. creating an empty file or making a copy of another. The second optimization is similar, but instead of the last cluster, it remembers the cluster where the last I/O took place. This is useful for reading the whole file in linear instead of quadratic time with respect to the size of the file. The first optimization works independently from the second one, while the second optimization can still be very useful for forward-going non-sequential I/O (i.e. when reads and writes are mixed with forward seeks) as the linked list traversal will start at the remembered cluster and not at the very beginning of the linked list. Of course, this second optimization will not be that much helpful anymore when multiple clients do sequential I/O at different offsets in the file, but let's face it (I am now going to derogate this issue again), this is not a common use case for a FAT file system anyway. When I wrote that both optimizations remember some cluster, it is also possible for both of them to forget that information under some circumstances, so don't forget these are mere optimizations that will help most of the time, but not always.

Below is a table and a chart that present some data I have collected in order to measure and visualize the expected improvement. I basically did a couple of experiments using a 10M FAT file system created using the mkfat HelenOS utility. For file sizes 1, 3, 5, 7 and 9 megabytes, I tried to measure the time to create a file of that size and also the time required to read the same file in its entirety. All times are in seconds and the commands I used to generate the desired workload were something like mkfile --size 5m /data/test and cat /data/test, respectively.

	Write A1	Write A2	Write A3	Write B1	Write B2	Write B3	Read A1	Read A2	Read A3	Read B1	Read B2	Read B3
1MiB	6	6	6	8	8	9	5	5	5	9	9	9
3MiB	21	22	24	40	41	42	18	20	17	38	37	38
5MiB	36	35	34	94	90	101	29	28	28	106	99	101
7MIB	51	52	51	219	202	230	39	39	39
9MIB	66	69	67				53	53	52

Note that I did all my experiments using Qemu and HelenOS mainline revisions 623 (Reads/Writes B1 - B3) and 624 (Reads/Writes A1 - A3). As you can see, for most file sizes I did three repeated measurements while for some combinations of file size and operation there is no data. The lack of data after Write B3 for a 7MiB file is caused by a reboot I was forced to do by my wife who started to threaten me as it was getting late and I was still behind the computer instead of being in our bed. I tried to resume my tests after the reboot on the other day, but I was not able to repeat the results from the previous day, so I rather stopped the experiment there avoiding non-consistent data (the new measurements were consistent with each other, as were the older measurements, but unfortunately the two groups of measurements were not consistent together). I think I could not repeat the previous day results simply because of a different load on my host system where the Qemu simulator with the experiments was running. By the way, I never said I was going to do a scientifically correct experiment, ok? And when I am at it, the times measured need to be taken with a grain of salt as I basically used a slightly jerky stopwatch provided by my Linux box.

Knowing the weaknesses of my measurement method, I still think I got some usable and persuasive results as can be seen in the chart above. It clearly shows the quadratic behavior of both operations in revision 623 and the linear behavior in the fixed revision 624. Note that the lines were added and smoothed by the chart drawing software to better illustrate the trends. The points in the chart correspond to the values found in the table above.

Being politically correct when talking to Jakub

2010-07-23T14:47:00.003+02:00

So you happen to be in a situation which involves talking to me. That is fine. I like to talk to people. There are, however, certain phrases that can easily alienate you as a speaker from me during a dialogue. If you use them you lose all my sympathies and you are basically done. Specifically, these include referring to the Czech Republic as to Eastern Europe or a post-communist country. While the latter simply feels annoyingly outdated 20+ years after the communists have lost it here and is a similar faux pas as referring to Germany or Austria as to post-Nazi countries (now or in 1965), the former is plainly wrong on all scale.

Geographically speaking, Czech Republic surely belongs to the western half of the continent. Looking on the map, the geographical border between western and eastern Europe is somewhere on the line connecting Finland in the north and Greece in the south, passing through Estonia, Latvia, Lithuania, Belarus, Ukraine, Romania and Bulgaria.

Historically and culturally, Czech Republic has been one of the countries in the Central European region. This was long before and now also after the Cold War, when the Iron Curtain and the communist orientation of Czechoslovakia put it to Eastern Europe. However this period was relatively short - only some 40 years - in comparison with the Central European cultural tradition of the country originating in the 17th century.

Politically, Czech Republic is no doubt in the western part of the continent as a full member of NATO and EU.

I prefer referring to my country as to a Central European country and am glad that some organizations, for example CIA, do the same. Unfortunately, there are also organizations, UN for instance, who get it completely wrong and do a favor to the increasingly self-confident Russian foreign policy. I am always ready to advocate for using the politically and geographically correct term Central Europe instead of Eastern Europe as I did and encountered partial success when BBC kept on using "Eastern Europe" in this context. On the other hand, I have recently heard some Czechs employed by one large American IT company to deliberately go with the Eastern European classification of their own country. I simply could not understand that.

Don't get me wrong - I think it is perfectly okay to refer to a geographically Eastern European country, such as Russia, Romania or Greece, as to an Eastern European country. But I can't stand when the term Eastern European is used inappropriately. When used in this way to refer to the countries that were once to the east of the Iron Curtain it has a very stigmatizing connotation which is very politically incorrect from today's perspective. Why should two similarly located Central European countries - Austria and the Czech Republic - be classified differently?

Prevention or post-integration fire-fighting?

2010-07-02T10:52:00.005+02:00

I've recently integrated fixes for the following four Solaris bugs:

6957730 cpuid_amd_getids() uses uninitialized variable coreidsz
6957751 nb_fini() leaks memory
6957783 nb_dev_reinit() leaks memory
6957795 Use-after-free in fm_nvlist_create()

which were found, along the following one:

6957739 rankaddr_to_chanaddr() may return undefined value

using the Coverity static checker.

The interesting observation here is that four of the above five bugs (only the use-after-free bug having a good excuse) would not most likely make it to the source gate at all, had the build process utilized all possible and available ways of checking for problems like that. In other words, it looks like that during the build process, we are deliberately suppressing some of the most useful warnings and errors such as those about uninitialized variables and code which has no effect. Of course, there is a feeble justification for it. The compilers are sometimes stupid and produce false-positives. So we rather silence those warnings and give no false-positive a chance to spoil our build logs that could stop our integration attempt.

The above bugs are mostly harmless as they appear in de facto unreachable code or in very rarely executed error paths, but two of my older blog entries (here and here) prove my case with real crashes caused by uninitialized variables.

And the solution? Utilize all the warnings, of course. Impossible you say? Certainly not. If you take a look for example at HelenOS, you will see its code base, say 150 000 LOC, being compiled with all warnings on (-Wall and -Wextra gcc switches) and all warnings turned into errors (-Werror gcc switch). Any warning thus has the power to fail the build, forcing the developer to either fix it if it is a real bug or work around it if it is a false positive. I am firmly convinced that working around the false positives is the bearable necessary evil, which many times outweighs the real evils described above.

Even though a time consuming task, enabling the warnings in Solaris (or at least in the onnv consolidation) and fixing what needs to be fixed in order to be able to build again will bring some long-term benefits, such as code quality improvements and software engineers not wasting their time by root causing and fixing bugs that were obvious and should not have been overlooked upon integration.

Qemu, OpenBIOS improve support for sparc64

2010-06-03T00:09:00.003+02:00

Qemu is firmly on the trajectory to become an emulator for everything. We are already using it to emulate amd64, arm32, ia32 and ppc32. It would not be hard to make use of its mips32 support too. Thanks to Igor Kovalenko, it very recently became the first open source emulator which can do pre-Niagara UltraSPARCs. The picture shows Qemu/sparc64 current with OpenBIOS current running HelenOS current:

P.S.: It looks like someone is already working on Qemu/ia64, so before too long, all HelenOS ports will be able to run on Qemu, which is just great!

Mixed declarations and code sucks

2010-05-31T21:46:00.006+02:00

C99 allows arbitrary interleaving of code and declarations inside one block. The previous versions of the language demanded that declarations are all at the beginning of the block, followed by the code which uses those declarations. Now, why on Earth did this change? When dealing with some eagerly C99-tified code such as unfortunately in HelenOS, I have to face several inconveniences:

the function looks ugly, not quite like the good old C, disturbed and it makes me feel like vomiting
if the function is a bit longer, it is difficult to find out where a variable is declared
likewise, it is much easier to declare a variable of the same name as is the variable which exists a couple of lines below
in general, it is difficult to keep track of all the variable declarations scattered all over the function's body
bigger maintainance overhead caused by moving the declaration up if it needs to be used by some code added before its declaration (think of a simple int rc variable)

Who said that a variable must be declared close to its first use, especially closer than is the beginning of the entire block? I am inevitably in deep disagreement with that person and want my good old-style declarations back!

Experimental version of Sysel integrated into HelenOS

2010-03-30T13:42:00.006+02:00

HelenOS now features a prototype version of the Sysel programming language interpreter developed by Jiri Svoboda. What advantage will Sysel bring to HelenOS?

In the first place, it is a language which was designed specifically for HelenOS (even though it runs on POSIX systems too), so there are no problems with meeting its build dependencies. The language has / will have similar features as some popular high-level languages like C#, but is incomparably easier to deploy on HelenOS. Just imagine the heroic effort and tons of bloat needed to port Mono to our system...

Additionally, using an expressive and high-level language like Sysel will enable us to write our system software on the appropriate level of abstraction, focusing on the business logic of the server processes themselves and avoiding low-level stuff like manipulating strings in C. As a consequence, the code should contain fewer stupid bugs and should take less time to write.

While we certainly do not plan to abandon C, Sysel will hopefully prove itself as a high-level language suitable for HelenOS system programming. The expectations are high, so let's see how the Sysel scene evolves.

Join #minix!

2010-03-15T22:30:00.004+01:00

No, I have not deserted to the worshipers of the old Raccoon. Instead, I have done something really bold which truly exercises one's courage and also tries one's survival instinct. On IRC, I entered the chat room used by the followers of the Herd of Gnus, also known as #hurd. Then I felt like visiting #minix too, so I entered the IRC command used to enter new rooms:

/join #minix

The problem was that I accidentally omitted the leading slash. This resulted in inviting all the on-line Hurd developers and followers to join perhaps their biggest rival (besides HelenOS :-), the MINIX 3 project. I realized my mistake soon enough and corrected myself so that I was not stoned. They actually reacted with something I would describe as slight amusement, which was good.

As an epilogue to this short story, I found out that my deed raised me an apprentice, who boldly repeated the same invitation today. Looks like they are slowly getting used to it.

The most modular ever

2010-02-04T19:57:00.003+01:00

This Tuesday I attended Lukas Mejdrech's defense of his master thesis Networking and TCP/IP stack for HelenOS system. Lukas literally did an excellent job presenting his work, which was credited accordingly by the committee.

One benefit of Lukas's thesis is that the same networking code can be built using either monolithic architecture or modular architecture. The former refers to a setup with one userspace server task handling the network interface and the network interface access protocol, and another userspace server task for handling ARP, IP, ICMP, UDP and TCP. The latter modular model has a separate server task for each network stack layer. While the monolithic design performs about twice as good, the modular has the potential to be more robust in case of programming errors. Lukas did some research and investigated the other common microkernel-based operating systems out there and found that all have a monolithic networking stack (e.g. MINIX 3 and its INET server). We are yet to merge the networking branch into our mainline and the TCP module still needs a little bit of coding, but with this in mind, HelenOS can be thought of as the most modular multiserver operating system on the planet and also the first to achieve the goal of coming with a fully modular unprivileged networking stack.

I like to draw diagrams on whiteboard

2010-01-19T23:33:00.006+01:00

Before implementing the unmount feature, I need to have a plan for the battle.

I could actually appreciate a way bigger whiteboard hanging in my home office, this small one is just a temporary fallback.

Why I am (not) in love with my car

2010-01-03T17:49:00.006+01:00

When it comes to cars, I think I am somehow attracting bad luck. Before my current car, I managed to basically destroy two cars, both in a similar way. That is not in an accident but by exploiting some hidden material fatigue. The first was an older Opel to which my driving on a high way burned a hole in one of the engine cylinders and the other was an older Hyunday Lantra which ran out of fuel at 150 km/hour and died. I must admit I knew I was going to run out of fuel and therefore I drove at 150 km/hour in a hope that I will make it to a gas station on the top of a hill. The irony is that I really made it there, but just before I took the exit from the high way, something cracked and irreversibly broke in the engine so that the car had to be scrapped as the repair would not pay off.

Then I bought my current car - a second-hand, but practically unused VW Polo. I have had it for two years now, but the list of issues I had with it looks more like twenty years. It all started when we were returning from a trip to Vienna. It was December 2007 (only three weeks after I bought it) and it was rather cold outside. The reason I mention the cold temperature is that it had its share in the extent of the damage that I am going to describe. I think we were already on a highway from Brno to Prague when we suddenly hit a hare. The animal must have been killed instantly, but it managed to blow out the front left fog light and the surrounding overcooled plastic grid and the fender. The repair cost me EUR 300, but only because I opted for non-original replacement parts. The original ones would have cost five times that much. Then I had a series of parts malfunctions: replacement of two ignition coils (EUR 150), switching device for the rear brake lights (EUR 42), oil pressure pump seal (EUR 33) and, finally, the servo unit (EUR 720) which broke on the Christmas day 2009. After the two ignition coils had died, I took the car to a periodic check (EUR 416). Between the switching device and oil pressure pump failures, my left rear view mirror was knocked off while waiting on an intersection by a truck in the neighboring lane (EUR 104).

Each month, I have driven about 1600 km only by commuting to work and back. One may therefore think that I have been using the car quite intensively, but still, I have the feeling that the car is causing me troubles intentionally. I am curious what will be the next malfunction and thrilled about the time it shows. The car is like a vampire - it just sucks money instead of blood.

Reincarnation server for HelenOS?

2009-11-05T20:04:00.005+01:00

Some operating systems have the capability to restart a server process after it unexpectedly turns up its toes. In MINIX 3, there is the reincarnation server (RS) to restart dead or hung services. In Solaris 10, the Service Management Facility (SMF) can do basically the same thing (see 6533008 for more on the planned feature for detecting unresponsive services). Now, when asked whether a similar feature should be implemented in HelenOS, I have to say: "No, thank you!".

Besides my belief that these reliability features may lead to system software of poorer quality because the software writers will simply rely on the restart, I have also seen too many bugs with the potential to render the system unusable especially thanks to the restarting.

In my case, these were all bugs in the start-up phase of the fault management daemon (fmd), which I happen to be sustaining as part of my job. You can find similar reports here by searching for "fmd full". The problem with fmd is that when it starts, it tries to process a backlog of unprocessed events, which is a non-trivial task. As with any complex tasks, there inevitably are bugs. So when fmd hits a bug in this phase, it dumps core and SMF restarts it. Then fmd starts to process the unprocessed events again and it dumps core again. This goes on and on until the core files fill up the /var file system and the customer calls Sun.

What a productive day

2009-10-21T17:29:00.007+02:00

Today looks like one of those more productive days. In the morning, the ONNV gate opened for me so that I was able to push two FMA fixes:

6883092 mem scheme's fmd_fmri_unusable uses the err1 variable in a wrong way
6889457 Bogus error messages can be generated when both 'offset' and 'physaddr' are present in a mem FMRI

In the afternoon, I switched from the fix integration mode into bug finding mode and found and filed the following two x86 VM bugs:

6893736 htable_reap() always reaps only 10 page tables
6893744 ptable_alloc() does not track active_ptables reliably

To be honest, I found these two because I simply had to find something VM-related in order to justify integration of a low-priority bug that I found some time ago:

6835500 Superfluous ASSERT(ht->ht_lock_cnt == 0) in htable_release()

Sometimes things cannot be done and bugs cannot be fixed in a straightforward way, but this time the extra effort and the bug-hunt payed-off.

Finally, I decided to create the following HelenOS ticket:

#136 Autodetect support for SYSENTER on ia32

because I have been thinking of fixing it for some time now. Unfortunately, I won't be able to work it this evening, because I am going to a pub with the folks from Prague's Solaris RPE to do some Pivo (Beer) Quality Assurance.

Cannot play MP3, but can ping localhost

2009-10-04T23:47:00.006+02:00

I have just updated my working copy of the HelenOS networking branch, which is now being exclusively hacked by Lukas Mejdrech, built it, booted it and... pinged localhost! To me, this is a real break-through comparable to the moment when HelenOS read from a file for the first time about two years ago.

Next Generation Jermar

2009-09-17T11:11:00.004+02:00

The Next Generation Jermar is called Víťa (short form of Vítězslav) and was born on September 16, 2009 at 3:17 AM.

The picture of Vita and his happy mother. Welcome to our world, Vita!

Great technical read for clock-cycle tuners

2009-09-10T12:40:00.005+02:00

Yesterday I found a great paper by Vlastimil Babka and Petr Tuma, both from Department of Software Engineering Faculty of Mathematics and Physics, Charles University (yes I am feeling very proud of my alma mater now :-). The paper is titled Investigating Cache Parameters of x86 Family Processors and via sophisticated benchmarking and use of hardware performance and event counters provides gory details about the internal structure of contemporary Intel and AMD processors' TLBs and memory caches, including precise estimates of the respective miss penalties in clock cycles. What is so cool about the information provided in the paper is that it is not publicly available from elsewhere (including vendor documentation) and that the estimates are provided for the individual parts that form the TLB and the cache.

Although the paper itself concludes that the presented data is probably too low-level for general software development, I can imagine that for example the L4 people may be interested in using it to update their 1993 paper Improving IPC by Kernel Design with todays clock cycle counts.

Who is holding this RW-lock?

2009-08-21T11:08:00.003+02:00

In Solaris, the kernel Readers-Writer locks are instances of the rwlock_impl_t type. The type is defined like this:

        typedef struct rwlock_impl {
                uintptr_t rw_wwwh; /* waiters, write wanted, hold count */
        } rwlock_impl_t;

As you can see, it is a mere 32-bit or 64-bit word, depending on the platform. Solaris uses all bits quite extensively. The lowest three bits indicate whether there are threads waiting for this lock, if some of them wants it for writing and whether the lock is held by a writer. The remaining 29 or 61 bits are used either to hold the address of the single writer holder or the number of readers already in the critical section. The reason why 29 or 61 bits are enough to hold a thread address is that thread structures are guaranteed to be aligned on at least 8-byte boundary, so the 3 missing bits will always be zero.

Now, while it is straightforward to determine the address of the thread who owns the lock as a writer, there is nothing like a list of threads that own it for reading. Which is worse, the current implementation does not even provide means to determine the address of a sole reader owner in cases when the lock is locked by one thread for reading and the next thread wants it for writing (and all consecutive threads, no matter whether writers or readers, are blocked until the sole reader exits the critical section). No need to say that exactly this kind of information may play an essential role when analyzing a crash dump of a hung machine.

Some time ago, I was involved in analyzing a hang of over 200 threads out of which the majority blocked on several RW-locks that were all held for reading:

6842265 lots of threads transitively waiting for scsi_flag_nointr_cv in scsi_transport()

In my analysis, I manage to identify all the reader holders of those locks using a technique, which is unfortunately not universal, but can be repeated in other similar cases. In my case, the data structure is a device tree, where each device node has a RW-lock. Besides a suitable data structure (e.g. a tree), the essence of success was also the knowledge of the functions that got hung: I knew that some of them will always attempt to read-lock a parent node and one of its children. Thus, when I found a thread waiting for e.g /pci@5c,600000, at a certain offset in dv_find(), I knew that it must be the thread which locked the root node for reading. To see how I identified the remaining reader-holders, I encourage you to see the above mentioned CR.

Synchronous vs. Asynchronous, Part 1

2009-07-26T13:18:00.016+02:00

Asynchronous Communication Using Synchronous IPC Primitives is Stefan Götz's master thesis that I have been recently reading through. In his thesis, the author designs asynchronous communication on top of L4's synchronous model. I think this paper is rather unique because it is the only L4 thesis (that I know of) which attributes some merit to asynchronous IPC. Other relevant L4 papers seem to neglect asynchronous communication with the typical microkernel-done-right dogmas.

Reading this thesis, especially the introductory chapter, was like pouring water on my microkernel antidogmatic mill because it clearly reveals a major drawback of purely synchronous IPC: increased number of address space switches. The rationale behind this claim is that with synchronous IPC, two address space switches must happen per one IPC request. The first happens on send, during a switch from the sender to the recipient and the second takes place on reply, during a switch from the recipient to the original sender. This happens per request due to the inevitable synchronous rendezvous of both communicating parties. If the communication protocol is a little bit more complicated, there will likely be several consecutive IPC sub-requests made by the sender and thus twice that number of address space switches.

Now, why there is not going to be so many address space switches with asynchronous IPC? Simply because the sender and the recipient do not have to wait for each other during each IPC. The message is stored in the recipient's mailbox without the need to switch to the recipient. Of course, in case of one IPC request, the number of address space switches will eventually be the same as the recipient needs to be switched to to read the message and the sender needs to be rescheduled in order to read the reply. However, in the case of more complex protocols, both the sender can push all sub-requests to the recipient's mailbox and the recipient can drain the mailbox without any additional address space switches. The asynchronous IPC will start to have diminishing returns as the dependency among consecutive sub-requests increases and the sender needs to wait for the results of previous requests prior to making new requests (and the communication becomes de facto synchronous).

The increased number of address space switches is a problem predominantly on processors that do not have tagged TLBs such as all IA-32 architecture and AMD64 architecture processors. The TLB, standing for Translation Look-aside Buffer, is a fast cache of recently used virtual-to-physical memory mappings. Without TLBs, the computer industry would probably have taken a different direction because the use of virtual memory would be so expensive (just imagine at least N physical memory accesses per page table walk incurred by a single instruction fetch on architectures with N-level page tables) that no one would like to use it (I think I have this observation from Andrew Tanenbaum's Modern Operating Systems, but my memory is weak so I might be wrong). With non-tagged TLBs, each TLB can contain mappings for only one address space, which has the unpleasant implication that non-tagged TLBs must be flushed on an address space switch. Flushed TLBs are gradually repopulated with the new address space's working set, which takes some time. Until the working set is completely restored, the TLB is not fully utilized and the system performance suffers the cost of all the page table walks.

The preceding paragraphs combined speak against massive and zealous use of synchronous IPC. The L4 developers came up with several architecture-specific tricks how to work around this problem. For example, in Improved Address-Space Switching on Pentium by Transparently Multiplexing User Address Spaces, Jochen Liedtke describes a way how to switch "small" address spaces without flushing the TLB on IA-32. No need to hold the surprise any longer, for address spaces of certain small size, they use segmentation. On AMD64, though, the segmentation has been practically removed from the processor, so this trick cannot be used.

All in all, it looks like asynchronous communication does indeed have some advantages over the synchronous communication. If taken ad absurdum, it is possible to devise IPC protocols which will linearly increase the number of address space switches with the number of requests made using the synchronous IPC.

Christmas time in June

2009-06-22T21:18:00.000+02:00

If you have followed the HelenOS development mailing list lately, you might have come to a conclusion that it must be Christmas again.

It all started with me committing support for non-root file system mounts. As the name suggests, this feature allows people to mount more than one file system at a time and to create a file system hierarchy as we know it from other operating systems. With this feature, it was finally possible to boot off a FAT file system ramdisk and mount an empty TMPFS file system under /tmp. Actually even better is booting off a TMPFS file system ramdisk and mounting a FAT (or any other) file system from permanent block device. The design is such that the mount point data is not stored centrally in e.g. the VFS task, but is distributed across the endpoint file systems. Each node in the endpoint file system server is a potential mount point. When something gets mounted under it, the mount point data containing an open connection to the mounted file system server is recorded in it. I was quite surprised and delighted when I found out that homogeneous mounts (e.g. mounting FAT on FAT) do not need any special treatment other than a little bit of locking discipline in the kernel when cloning the VFS connection to the mountee for the mounter. The figure below depicts how VFS and the endpoint file systems are interconnected (displaying only connections along which file system requests can be sent).

Non-root mounts are essential not only for mounting file systems on real block devices, but also for seemingly unrelated features such as console inheritance between e.g. the bdsh shell and a newly spawned task. This is where Martin got involved. He added a new file system called DEVFS, which roughly speaking associates file system nodes with IPC services such as console. Each virtual console has a corresponding node under /dev (for instance /dev/vc0). When spawning a new task, the shell application passes it its stdin, stdout and stderr. When the application writes its standard output, the write() request first goes to DEVFS, which in turn sends it to the console server.

Jiri spotted the opportunity and created three block device drivers: gxe_bd, ata_bd and file_bd. The first one is to be used with the arm32 and mips32 ports on the GXemul simulator. It controls the simple GXemul disk device. The second one is a simple driver for the ATA disks and can be used in the Qemu simulator with the ia32 and amd64 ports. Note that Jiri now warns against using it on real systems with real disks. The last block device driver, file_bd, is a loopback block device. It creates a new block device backed by a file. If the file contains an image of a file system, it can be mounted as though it was located on a real block device.

In parallel with the trunk changes, Lukas Mejdrech committed his IP bits to the networking branch. These changes allow HelenOS running in the Qemu simulator to receive data from the web browser running on the localhost. True, it still cannot answer the HTTP request (for that there would have to be more of the TCP module and some simple HTTP server), but it nicely demonstrates the progress.

To give this blog some real meat and to show that there are problems too, consider the following diagram:

It illustrates the sequence of requests made when trying to mount a file system image stored in a regular file located on an already mounted TMPFS file system. What is notable is that there is a cycle in the diagram, because FILE_BD is both a block device driver and also a client of the VFS server. For some time, there has been exactly one open connection between VFS and each endpoint file system, and one connection fibril to handle it. So what do we make of it? It's a clear deadlock situation: the sole fibril will be waiting for answers to requests made along the cycle while VFS will wait for the same fibril to process a new read() request.

A similar situation occurred when one task blocked in read() waiting for a character to be read. This put the sole DEVFS fibril to sleep and delayed consequent requests to write() a character on the remaining virtual consoles.

It took us some serious thinking before we realized that we are hitting exactly this problem. Eventually I fixed it for now by allowing VFS to open multiple connections to the endpoint file systems.

So to sum this writing up, the next upcoming HelenOS release will be full of (hopefully well debugged) new interesting features.

OpenSolaris 2009.06 is out!

2009-06-02T10:31:00.005+02:00

The new release of OpenSolaris was released yesterday and if I count right, it contains two fixes of mine:

6747539 hati_pte_map() should undo HTABLE_LOCK_INC(ht) on LPAGE_ERROR
6743475 Some MCA banks are skipped too fearlessly

I have already blogged about the first fix in one of my previous blogs. The second fix slightly improves OpenSolaris MCA (Machine Check Architecture) code by adhering more closely to the Intel and AMD specifications and, by virtue of that, supporting more error detectors.

Idea: MFF UK almanac

2009-05-22T11:35:00.004+02:00

A couple of years ago, I had a great idea of putting together the greatest software created at my alma mater - Faculty of Mathematics and Physics, Charles University in Prague (MFF UK) - and releasing it on a CD in the form of an almanac. At that time, the idea was presented to the academic senate by my complice Martin Decky where it received some positive feedback. Unfortunately, the idea died shortly after that due to the lack of further momentum and support.

Nevertheless, it still seems to me worth doing even on a longer than yearly basis. So what is the general characteristics of projects created at MFF UK that I have on my mind?

Most of them would be fruits of a subject called Software project. During my university years, it used to be a multi-semester enterprise, which usually spanned several years and involved from 4 to 7 students. Projects like Lestes (C++ compiler), HelenOS (operating system) and Ultimate Racer (car racing game) would be good examples. Most of these projects were abandoned by their authors shortly after they had received credits for the subject, and were never finished.

Another group of projects are random cool projects that I know of. From the older ones, these are the ECCE C compiler and the T3 operating system and its distributed successor T4, all produced by the Ulita group. Nowadays, I am afraid, you can find only technical papers about these systems. From the new ones it is Mikulas Patocka's SPAD operating system and its file system SpadFS.

Thanks to my personal bias, the almanac would be mostly composed of operating systems and compilers (already five in total), but I would leave some space for other projects such as the Links web browser, the Ronja datalink or the AA-lib graphics library. Of course, I don't know all the cool MFF UK projects out there, so if you have a tip, feel free to post it to the comments.

Time to go embedded

2009-04-22T15:28:00.005+02:00

It looks like we will need to optimize HelenOS for embedded systems because it is getting too big for some machines. One particular machine, which has been causing us problems is the Simics Serengeti system, a close relative of this half-ton UltraSPARC III-based baby. The problem which puts Serengeti into the embedded category is that its OpenFirmware is for some reason unable to allocate enough contiguous space to accommodate the HelenOS boot image and/or the ramdisk. One therefore needs to be somewhat picky when configuring the system for Serengeti, which results in a reduced set of features that can be used on this system.

I am planning to change the build and configuration system in a way which would support optimizing for size. This ranges from using -Os instead of -O3 and stripping the binaries to cherry-picking what binaries will be part of the ramdisk and deploying the hopefully-soon-to-be-completed dynamic linker.

Source of interesting Solaris bugs

2009-04-01T15:03:00.006+02:00

In one of my previous blogs, I talked about how I accidentally discovered two new VM bugs in Solaris 10 when I had done regression testing of my own fix for some unrelated problem. A couple of weeks ago, I found myself in a similar situation. This time, I did regression tests for a Solaris 10 backport of one of those two VM bugs (I had to fix it in Solaris Nevada first) and one of the tests resulted in kernel heap corruption. I was lucky, because the crash occurred on a debug kernel which runs with the kmem_flags variable set to 0xf. Besides other things, this setting enables buffer transaction logging and detection of writes past the end of a buffer. With this aid and the crash dump, I was able to root cause this bug. Exactly as in the previous case, the culprit here was an uninitialized variable. If you are interested, have a look at the following CR:

6823941 sendvec_chunk() uses uninitialized variables

What is striking is that neither of these uninitialized variables was discovered in compile-time or lint-time. Well, it's not that striking when we notice what options are used to build Solaris. Among others, the Solaris compiler wrapper utility turns on -Wno-uninitialized, which explains why GCC does not complain.

The morale of the story is threefold:

do regression testing of debug binaries too or they become source of fun for someone else
do not write functions that span over 14 standard terminals in the height dimension
compiler warnings can be useful

How big is HelenOS? How small it can get?

2009-03-19T16:39:00.005+01:00

After some discussion with Jiri, I thought it would be useful to know how big is the HelenOS kernel. I've done several measurements and obtained interesting and surprising results. For each architecture, I have built the system using the default configuration, and a configuration with all options disabled. Both these builds would use the -O3 optimization. Then I changed the -O3 to -Os and rebuilt the minimal configuration. On some platforms, the -Os could not be used right away due to missing parts of the softint library, so I used -O2 instead. The HelenOS build system does not strip the kernel.raw binary so I did that manually and wrote down the final size obtained. These are my results:

What is immediately clear from the chart is that there is something wrong with amd64 as it gives a kernel which is about three times as big as the other kernels. We'll clearly need to investigate this.

All 32-bit kernels and all 64-bit kernels except the amd64 kernel are comparable in size. The ia32 kernel is the smallest one, even though it would be interesting to know how small would be the arm32 kernel with the -Os optimization. When the minimal configuration is optimized for size and stripped, the ia32 binary is only 116 KiB. When I alter the kernel Makefile, I can get additional 4 KiB away by not using frame pointers (-fomit-frame-pointer). Even though not supported right now, further 27 KiB can be shook off by not including the .bss section in the kernel image. This will give us quite respectable kernel size of just 84 KiB! This 84 KiB kernel would be completely headless (almost no drivers compiled in) and trimmed down (e.g. without SMP support), but it would do its job well. I believe further improvements are still possible, but the continued effort will most likely start to have diminishing results soon.

Interfacing with the user in HelenOS

2009-03-13T18:11:00.021+01:00

For about a month, Martin, Jiri and me have been torturing HelenOS sources in a distributed attempt to improve the subsystems responsible for processing user's input and output. Things are still not perfect now, but before these changes, this part of HelenOS was a complete disaster. Here are few examples of how badly designed this part of HelenOS was:

There was no layering between the hardware drivers and the code which interpreted characters received using these drivers - everything would happen inside the driver itself. With this setup it was not possible to support different hardware configurations in a clean and generic way. For instance, the ns16550 driver assumed a Sun keyboard attached to the ns16550 serial port. This was, of course, something which complicated the use of the driver for plain serial communication between two computers interconnected with a serial cable.
Neither the kernel nor the userspace drivers supported more than one instance of each character device (e.g. i8042, ns16550, z8530).
Because of only one instance was supported, each driver inferred its role as stdin or stdout without asking.
The way how interrupts were sent to userspace device drivers required a little brother kernel driver for the same device which would accept the interrupt and send it to the userspace server.
Both kernel and userspace drivers were rather platform specific, either using memory mapped accesses to device's registers or separate I/O instructions.
There was also some duplication of code, both on the physical device level (e.g. duplicate ns16550 driver) and also on the character interpretation level (e.g. several occurrences of code which processed serial line input).

Jiri was the first to do something about it. He focused on the userspace kbd server and introduced layering into it. In his design (see the picture below), there are port drivers, that control the physical devices such as the i8042 or z8530. In combination with my later changes, the port drivers use generic PIO operations to directly interact with the device. Each port driver would typically register an interrupt handler for the device's interrupt and provide an interrupt top-half pseudocode. Upon interrupt, the port driver reads data from the device and pushes it to the next layer. The next layer in this case is the controller layer. It assumes a PC keyboard data on input. If the input data does not come from a PC keyboard but, for example, a serial line, the controller layer driver transforms the stream of ASCII characters into PC keyboard scan codes using a scan code simulator. This layer would also convert other forms of input such as that coming from the Sun keyboard to the set of PC keyboard scan codes. Thus, the PC keyboard scan codes form a common representation of data coming from the lower layers. In order to support multiple keyboard layouts and to distinguish physical location of the pressed key from its label, the common representation is referred to as to key codes rather than scan codes. Key codes are further pushed to the next layer called the event layer. The event layer keeps track of the status of the Shift, Ctrl, Alt and the Lock keys and performs the layout translation. So far, two layouts are supported:

US QWERTY
US Dvorak

After the layout translation is done, the key event is sent via IPC to the only consumer of kbd - the console server.

Encouraged and inspired by Jiri's success, I wanted to fix the layering and all the other above mentioned problems in the kernel too. I started by converting the kernel drivers to the PIO (Programmed I/O) interface. The PIO functions abstract the implementation of the machine's I/O space away and allow the drivers to be written in a generic way, regardless of the fact whether the device is in a separate I/O space or is memory mapped. Once I had PIO for all platforms, I started to convert all character device drivers to it. That was probably the easiest part. In parallel, I began to slowly move away from the one-instance per device driver model and free the kernel drivers from the duty to notify their userspace counterparts about interrupts via IPC. That was probably the hardest part as it required:

extend the interrupt top-half pseudocode to support independent userspace drivers, and
rewrite the way how interrupts are dispatched in the kernel, and
fix the userspace drivers to use the new pseudocode.

Extending the pseudocode was rather fun. I took the chance and fixed the existing pseudocode to only make use of PIO when accessing the device and added operations for testing bits, conditional execution of blocks of pseudocode commands and accepting the interrupt.

Rewriting interrupt dispatching looked like a real teaser to me because the kernel interrupt structures are never deallocated while the userspace interrupt structures need to be allocated and deallocated dynamically on demand. It was also interesting from the synchronization point of view as I didn't want to deallocate a userspace interrupt structure while the interrupt is in progress. In the end, I came up with a solution with separate hash tables for userspace and kernel interrupts. My change broke some things such as klog and kconsole notifications as well as switching between kernel and userspace drivers. The latter used to work thanks to the grab and release methods in each kernel driver. But since the kernel driver became a distinct entity and these functions were removed, the driver toggling had to be solved in another way. This is were Martin got involved and implemented a clever fix:

if the silent variable is true, search the userspace hash table first and the kernel hash table second

if the silent variable is false, search the kernel hash table first and the userspace hash table second

This gives a userspace driver a chance to process the interrupt if the kernel console is inactive, but if there is no userspace driver to claim the interrupt, the kernel can still react to it, and vice versa.

Having freed the kernel drivers from the burden of interrupt notifications for userspace brothers, I could finally proceed to fix the other problems and layer the kernel input subsystem into several components. At the lowest level, there are serial controller drivers, much like the port drivers in the case of the userspace kbd server. Each driver connects either to a keyboard input module or a serial line input module. Contrary to the userspace server, the kernel input modules convert raw data from the serial controller drivers to a stream of ASCII characters and feed it the connected component, which is most likely the kernel console.

After my change, Martin noticed that the data structure which has been used to pass characters between various components - chardev_t - is bidirectional in nature, even though it is mostly used for one direction only. After some 30 commits, the whole kernel and all drivers and input modules were converted to use indev_t for input devices and outdev_t for output devices instead.

On the userspace side, the kbd server is still a monolithic piece of software with a limited hardware support and flexibility configured during compile time. We are considering changing this towards a more modular scheme in which there would be one running port driver task per device instance. That would allow us to move the configuration from compile time to runtime.

Interesting discovery about HelenOS/sparc64

2009-02-14T17:18:00.007+01:00

In the newly released HelenOS 0.4.0, we deliver improved support for 64-bit SPARC machines. This improved support is not so much interesting because of the higher number of peripherals supported (even though that number has also gone up), but mainly because of the higher number of individual SPARC processor models on which HelenOS can run. When I go through the processor list on this page, and count the UltraSPARC T1 processor, which we will soon officially cover too, I inevitably come to the conclusion that HelenOS has the best support for SPARC V9 processors, right after Solaris, OpenBSD and Linux.

First contact pictures

2009-01-30T09:45:00.008+01:00

Over the time, I collected photographs of several exotic (i.e. non-x86) machines booting HelenOS for the first time. Below you'll find pictures of SGI Indy, Sun Ultra 60, Dell PowerEdge 3250 and iMac G4 all booting HelenOS in some point of its evolution. Some of the pictures are in a very poor quality as they were taken using a cell phone.

The first picture was taken by Ondrej Palkovsky and depicts the early mips32 port running on SGI Indy. The Indy support has been removed from the trunk some time ago as part of some cleanup, but continues to live in our memories and, in the first place, in our repository history. I believe the picture predates the userspace support in HelenOS and therefore you can only see the kernel having passed some sort of test followed by the kinit kernel thread printing "kinit..." in 1-second delays over and over again.

The next picture illustrates one sparc64 email debugging session attended by me and Martin Decky. I believe the picture shows the first successful boot on Sun Ultra 60 in Martin's office. The picture was taken using a cell phone when Martin wanted to share his view with me. Note that at that picture, the visual for the Ultra 60 framebuffer is still wrong and the kernel panics due to a bad fast_data_access_mmu fault.

On the next picture taken and sent by Jakub Vana, you can see one of the first successfull boots on a dual Itanium II system. Even though the ia64 port was long able to run entire userspace in the Ski simulator, the real world Itanium was something quite different and required additional work to reach a comparable level of usability.

The last picture (so far) motivated me to write this blog entry. It's only several hours old and shows an astonishing progress on a real-hardware support for the ppc32 port. It was taken by the same Martin in the same office, I believe, using the same cell phone as the sparc64 pictures. HelenOS/ppc32 runs in the PearPC simulator, but seeing it run on real hardware after lots of failed attempts was very rewarding, even though the full credit goes to Martin.

How the Russians renamed my wife and confused the French

2009-01-13T15:10:00.003+01:00

Sometime in October 2008, my wife's Russian passport was about to expire. Because of that, she visited the Russian embassy in Prague in order to get it renewed. Because you never know what documents and in what form will the Russian authorities need (and how much it will cost), it took four visits in total: Marina initially forgot to bring a couple of her photographs with her, on her second visit they decided they needed our wedding certificate again (they've already had it since after our wedding when they official changed her name to Jermářová) and on her third visit, she successfully submitted the passport renewal form plus a negligible fee of CZK 3000, at that time an equivalent of roughly EUR 125 or USD 176. On the fourth visit, it was the embassy's turn. Because perhaps it would be considered a bad form in Russia to let the holder of the newly issued passport verify if everything is alright with it, they let her sign the completion certificate and let her go with the precious document. What a surprise when I found out that along with a new travel document, she has also received a new name! After the marriage, the old passport was altered to spell her new last name Jermářová (both in the Cyrillic and the Latin alphabet). In Cyrillic, the closest transcription from Czech would be Ермаржова (which translates back to Czech as Jermaržova), while the preferred transcription to mere Latin alphabet is simply Jermarova. Now, the problem is that in the new passport, they used a different transcription which is neither Czech nor mere Latin alphabet. It reads as: Ermarzhova, which is quite far from what I have in my passport: Jermář, and can confuse people with lack of imagination.

I suspected future problems, but the Czech authorities, despite all worries, did not have any problems with this and continued to call my wife using her Czech last name. They even managed to fix the name on the Czech/EU permanent-stay enclosure in the Russian passport, so I hoped that it is safe to travel with Marina at least across Europe. In December, we went to Paris and spent few days there. Right before our home flight, the French airport authority, who served at the check-in, diligently and compulsorily identified the name mismatch and called a more knowledgeable colleague to help. It took me by surprise that the French authorities, the true candid Europeans, would not consult the EU enclosure of the passport, but would accept the bigger challenge and try to decipher the national page which is half in Cyrillic and half a Latin transcription from Cyrillic. After some explanation from my side, the French were eventually able to find some lexical similarity between the two last names (the Levenshtein distance between Ermarzhova and Jermář is 7 and 4 between Ermarzhova and Jermarova).

Time for a new survey

2009-01-04T15:30:00.004+01:00

Two months have passed since I published the microkernel survey. The survey is now closed, but I saved the results here. Even though interesting, so few people took part in it that it does not reveal any trends. I need to give a serious thought to how to attract more readers to my blog. I can either improve the contents or have better surveys :-) Hopefully I'll have a little more visitors after the next release of HelenOS, which is still not here, but is definitely approaching.

This month, I decided to let you vote for your favorite processor architecture. There is nothing tricky in that question. If you miss your favorite one on the list, just feel free to post a comment and tell me what you think. I never vote in my own polls, but if I did, I would have voted for ia64 and sparc64...

History of the HelenOS repository in 230 seconds

2008-12-30T11:52:00.010+01:00

Last night I played with code swarm and generated a short movie depicting the history of the HelenOS project repository in 230 seconds. The video below utilizes four frames per day. I have obtained slightly different results with one frame per day and also eight frames per day, but this one seems to be the best.

Things get interesting around the times when there was a HelenOS camp and also when people worked intensively on a branch as part of their University assignment.

Update: Here are download links for the one-frame-per-day video, four-frames-per-day video and eight-frames-per-day video.

Debugging HelenOS with QEMU and GDB

2008-11-26T21:52:00.004+01:00

It is possible to use the combination of QEMU and GDB to debug live HelenOS system. You need to have the respective cross-GDB ready if you intend to debug the system on a non-native architecture. You start by running QEMU like this:

$ qemu -cdrom image.iso -s -S

The lower-case -s tells QEMU to wait for the GDB session at port 1234. The capital -S instructs QEMU not to start the simulation before you enter c from the GDB prompt.

Once you have QEMU waiting for a connection at port 1234, do the following:

(gdb) target remote localhost:1234

After that, you should see something like this:

Remote debugging using localhost:1234
0x0000fff0 in ?? ()
(gdb)

GDB is now ready and you can start the simulation, but you still don't have any symbols. GDB allows you to load the symbol information using the symbol command:

(gdb) symbol kernel/kernel.raw

Instead of the kernel symbols, you can load symbols of any of the userspace ELF binaries, but bare in mind that you can have only one set of symbols loaded at a time. Now you are ready to proceed and can start the simulation:

(gdb) c

Later you can break back to the debugger by pressing Ctrl+C in the GDB window.

This method gives you a nice debugging features for QEMU targets on which HelenOS runs. One problem is that the simulator cannot separate the execution of the kernel from the execution of the separate userspace tasks so if you single step long enough, there will be some context switches that you won't be able to filter out. In this light, the debugging approach seems to be most suitable for debugging the kernel.

Thoughts about observability

2008-11-08T12:30:00.013+01:00

My English language spell-checker tells me that there is no such word as observability. In the Solaris technical parlance, we use this term to refer to the software features that allow one to have a pretty good idea about what was the system doing sometimes in the past, be it in the time of a crash or in the history of the running system, and also what is the system doing right now. There are many observability tools and techniques, some more sophisticated than others.

For instance, Linus Torvalds, as well as a great deal of his fellow Linux kernel developers and also Linux users, expressed himself, and this is not an exact quotation, that real men use debug prints and their brain to debug something as complex as the Linux kernel. I last saw someone else to use these words yesterday on the Czech Linux site root.cz. Linus has also said, at least once, that Solaris is a piece of crap mainly because it has observability features that go far beyond the debug print-brain technology. According to Linus, the holy grail of debugging is:

So forget about it. The whole model is totally broken. We need to make
bug-reports short and sweet, enough so that random people can
copy-and-paste them into an email or take a digital photo. Anything else
IS TOTALLY INSANE AND USELESS!

Frankly, sometimes it is too early for the crash dump to be generated so we occasionally also receive digital photos of the panic messages from our customers or from the OpenSolaris community members, but you hardly ever get enough information from a photo. What you really want to have is a crash dump, because in 99.999% cases it contains all the state which lead to the crash.

In Solaris the debugger is called mdb or kmdb, depending on whether you are looking into a coredump, a crashdump, a running system or a running system in single-stepping mode. Solaris has had a kernel debugger for ages and for my job, it is the ultimate tool without which I would be condemned to looking into a crystal sphere^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H digital photos.

A colleague of mine, Vineeth Pillai, is sustaining Sun's SAM QFS product. Besides the Solaris modules, he also has some Linux kernel modules to take care of. Whenever there is a problem with the Linux kernel module, he is out of luck, because he doesn't get the usual Solaris crash goodies and must resort to the printk() workaround. True, there has been a kernel debugger in Linux since 2.6.26, but the customers are not running that version yet.

My last root-caused bug is a good example of usefulness of crash dumps. Last week, I worked on a bug which was occurring somewhere between OBP and the kernel. The usual observability tool - kmdb and dtrace - were not an option here due to the special restricted context of the OBP's callback_handler environment. Luckily, I was able to use a feature of Solaris debug kernels called TRAPTRACE. With this feature enabled, the kernel records essential data about each trap which occurs on the system into a kernel buffer. When the system later crashed due to CR 6750765, I was able to successfully reconstruct the flow of events that resulted in the crash using that data and mdb's ::ttrace dcmd which presents them in a human readable form. There was no way I could do without the crashdump and the TRAPTRACE data in this case.

I'd definitely like to see similar observability features in HelenOS one day. So far, we were using the simulator features to replace the role of the kernel debugger, but that is going to be unsustainable in the future, as the project focus shifts more towards real metal. The trend is looking good though, we already have the trace task, which can be best described as a variant of the strace command on Linux or the truss command on Solaris.

HelenOS Camp '08 Pictures

2008-11-03T22:08:00.006+01:00

The first picture was taken in a restaurant/cafe called Alenka. We chose Alenka because they make good coffee and have free wireless Internet access. It was Saturday morning and we were warming up before we'd climb up to the base camp.

The next picture shows the part of the camp with commit access in the base camp. We already have Vineeth amongst us, so it must be Monday afternoon.

This is not Yetti, but our Itanium guru Jakub.

Martin, most likely dealing with nonpoisonous snakes like Python.

And this is me, trying to put on some weight and FATten a little more.

And here's Vineeth, raising his ARM.

Netbooting HelenOS on Sun Ultra 60

2008-11-03T20:55:00.005+01:00

This is the setup for netbooting HelenOS on Sun Ultra 60 which worked for me. The image server is currently Ubuntu 8.10 Linux box.

Install and setup tftpd. This is what my /etc/inetd.conf looks like (make sure it remains on one line):
tftp dgram udp wait nobody /usr/sbin/tcpd /usr/sbin/in.tftpd -s /tftpboot

Install and setup rarpd; I assume that the Linux box will be on network 192.168.1.0 and the HelenOS/sparc64 box will be given address 192.168.1.2. In my case, the MAC address of the Ultra is 08:00:20:CD:BD:67. My /etc/ethers is as follows:
08:00:20:CD:BD:67 192.168.1.2
Create world-readable /tftpboot and copy the sparc64 image.boot there as file C0A80102. The number corresponds to 192.168.1.2 and is what the HelenOS host will be looking for at that directory.
Interconnect the Ultra with the Linux box using a switch or a cross-cable. Then power on the Ultra and when you reach the ok> prompt, type:
boot net

If something goes wrong, I suggest that you use tcpdump to monitor the network traffic on your NIC.

Big microkernel survey

2008-11-01T16:11:00.012+01:00

I have just started a new survey on this blog so that I can have a better idea of what features people value the most in microkernels. I hope to receive more feedback than for my previous survey which asked if people would consider contributing to HelenOS.

This time, I provide eleven different options. Because these are mere one, two or three-word labels, I'd like to give some explanation on what I meant by each. Hopefully, people will feel encouraged to comment their choices in the survey by attaching comments to this entry.

performance - some microkernels (e.g. L4 variants) seem to be focused on maximizing the IPC performance and reducing the number of different kinds of switches, sometimes the assembly is used as the implementation language;
robustness - some microkernels (e.g. Minix) have the ability to detect failures of system services. In case of a failure the service is restarted.
small memory footprint - microkernels tend to be smaller than other operating systems' kernels. As such, they take up smaller size of the computer's memory and can be cache-hot most of the time;
real-time features - RTOS are often implemented as microkernels;
observability - ability to find out what is/was the system doing; e.g. Solaris is very good in this;
component design - separation and isolation of individual functional components such as the file system services in HelenOS. The components can only communicate via a well-defined protocol;
desktop features - some microkernels (e.g. QNX or XNU) could have been used as a desktop operating system;
high-end features - things like SMP support, but also hot-swap, hardware fault management etc.
portability - the ability to run on multiple processor architectures and the ease of porting to a new architecture;
security - smaller trusted code base should be more secure by virtue of having fewer LOC and thus fewer security bugs;
standard compliance - compliance with POSIX or any other operating system API standard

Two interesting Solaris VM bugs

2008-10-31T10:13:00.012+01:00

I ran into two interesting virtual memory bugs while testing some bugfixes for our customers about a month ago. The problem was occurring on some older 32-bit machines when running debug builds of Solaris 10. When running one of our test suites, the system would hit an assert() and panic like this:

assertion failed: ht->ht_lock_cnt == 0 || ht->ht_valid_cnt > 0, file: ../../i86pc/vm/htable.c, line: 994

Searching our bug database quickly revealed the possible culprit:

6607917 assertion failed: ht->ht_lock_cnt == 0 || ht->ht_valid_cnt > 0

This is a bug, which was already fixed in Nevada, but not in Solaris 10. Moreover, it can only happen with debug kernels, so there is no need to worry about it in case of our production release, i.e. Solaris 10. Right?

Wrong. Besides the fact, that a test suite running with user privileges was panicking the test machines on which I and a handful of my colleagues did our pre-integration testing, I discovered, that this is certainly not 6607917. The bug requires two threads to be calling htable_release() on one htable_t structure at a time, while the crashdump I received from the PIT machine showed only one thread.

You can see the analysis of the crashdump in the description of the new bug I have filed for it:

6747539 hati_pte_map() should undo HTABLE_LOCK_INC(ht) on LPAGE_ERROR

In this case, the kernel detects a collision between a new large page mapping and the current contents of a page table. While this conflict is handled by the system, it forgets to unlock the respective htable_t structure when it bails out in hati_pte_map(). There is no way a debug kernel could not panic afterwards, because the very next call after the failed hati_pte_map() is htable_release() - the function where the assert is hit.

But wait, there's more to it. From my crashdump, I noticed that the htable_t in question was empty - there were no entries in the corresponfing page table, so how come there could have been some conflict between a new large page mapping and the base size old mapping? Impossible.

/*
 * Set the new pte, retrieving the old one at the same time.
 */
old_pte = x86pte_set(ht, entry, pte, pte_ptr);

/*
 * did we get a large page / page table collision?
 */
if (old_pte == LPAGE_ERROR) {
 rv = -1;
 goto done;
}

This snippet above shows how the collision is detected - via a return value from x86pte_set(). So maybe there is something wrong with that function and we should not have detected the collision in the first place. And really, there is a regression introduced into Solaris 10:

6747627 x86pte_set() uses uninitialized variable "prev" when PAE is not enabled

So the first bug was actually assisted by the second bug - a piece of uninitialized memory confused x86pte_set() which in turn signalized an LPAGE_ERROR, which caused hati_pte_map() to bail out too quickly, leading to a kernel panic.

I wanted to post about these two new bugs earlier, but I also wanted to use them during the CDA course I delivered last week in Prague Sun office, so excuse the delay.

New subject at Charles University

2008-09-29T15:55:00.006+02:00

It looks like I'll be teaching this course at Charles University in the summer semester. The course is basically an extension of the mini CDA course that I and my two colleagues delivered at MFF UK in April. It is also an extension to the internal Sun x86 CDA training, that I am going to deliver with my companions this October.

In addition to the mini version, the full-fledged university course will cover 32-bit x86 architecture too and compared to both the mini and the internal courses, we'll also venture into CDA on SPARC. We don't want people to suspect us that we are using this university subject to market Solaris based on its superior observability features, so the course shall teach people how to tackle crashes on Linux and Windows as well, even though at least in the Windows case we plan to invite an expert in that area.

Keeping the balance right

2008-09-03T21:35:00.004+02:00

In order to balance my previous blog, I decided to sum up some recent achievements on the HelenOS front. These things are well known to people who follow our mailing list, but I suspect that the majority of people out there are not even subscribed.

I scratched two of my long time kernel itches: the infamous "Sleep not implemented" panic and the kernel's inability to allocate memory from a low-memory region.

The "Sleep not implemented" panic occurred every time when the kernel invoked frame_alloc() in its blocking mode under very low memory conditions. That happened when the (virtual) machine had less memory than what the system required. After my modification, the offending thread would be simply put to sleep instead of panicing the kernel. When there is enough memory, the thread can complete the allocation request. The catch is that when all of the threads just allocate and none of them frees memory, all the threads will be put to sleep and none will run. You still have the possibility to break into kconsole and kill some of the tasks so that its allocated memory is freed and others can run. You are out of luck when the one which went to sleep is the kconsole thread itself (that could happen when running the kernel tests which allocate lots of memory). The situation can be further improved when we have support for paging pages out and back in. Btw, this improvement finally found good use for condition variables.

As mentioned above, the kernel can now theoretically allocate memory from the lowest 4G (contrary to allocating from all memory available). This is necessary, for example, for allocation of secondary processors' GDT on amd64. The fix works in the following way. When a memory zone is created, it is associated with flags that describe it. The frame allocator was slightly altered to only consider zones matching flags passed along with the request, if some were specified by the caller. What remains to be done is to change the way how memory zones are created in the architecture specific initialization code. In particular, it is necessary to avoid merging zones with incompatible flags.

My biggest achievement, however, is the read-only support for FAT16 as the root file system. Together with Tim Post's addition of a simple command line interface, Martin Decky's boot-from-ramdisk infrastructure and Jiri Svoboda's task loader, the system is now much more usable than before. The biggest differentiator is that you can now interactively walk the file system (either FAT or TMPFS) and spawn as many new tasks (located on the file system) as you want. Due to ramdisk size issues, the FAT filesystem is now usable only on ia32 and amd64, but this is likely to change soon.

My failure to implement &&

2008-09-02T23:30:00.007+02:00

I spent half of the past weekend implementing support for the logical AND operator for the Lestes compiler. I should say right at the beginning that I still haven't succeeded, but I decided to share the details of my struggle with you, the kind reader.

The state of the current && and || support is that the parser recognizes these constructs and generates semantic structures for them. The problem is that the part of the compiler which is supposed to generate the respective pseudo code is currently a mere assert() saying that these operators are not yet implemented.

The particular problem of implementing these two is their short-circuit evaluation. When you have, for example, A && B, you first need to evaluate A and if it evaluates to true, only then you can start evaluating B. Without this special feature of C/C++, I'd be able to implement && and || in half a minute, using the already existing template for binary operands. With short-circuit evaluation in place, I need to insert a couple of extra pseudo instructions in the right order:

psp:
psp(A):
L = evaluate(A)
nsp(A):
PI_BF(L, msp2) /* skip evaluation of B */
psp(B)
R = evaulate(B)
nsp(B)
DEST = PI_LAND(L, R) /* logical AND */
msp1:
PI_BA(nsp)
msp2:
PI_MOV(L, DEST)
nsp:

In the pseudo code snippet above, psp stands for previous sequence point and nsp stands for next sequence point. You can see that the sequence points figure as destinations for branch instructions PI_BF and PI_BA and that each expression is surrounded by the psp-nsp pair. nsp(A) and psp(B) are essential in that I must place the PI_BF instruction somewhere between the two, enforcing thus the order of evaluation to start with subexpression A. I added two extra sequence points msp1 and msp2 to help me to enforce order on the pseudo instructions in steps 9 through 14.

While I am quite successful in generating code for the end of the sequence, I can't figure out how to place the PI_BF pseudo instruction without generating an infinite loop. My guess is that somewhere in the Lestes code, something enforces the opposite order of evaluation of A and B. That would, of course, made the topological sort algorithm spin forever. I've tried to place the instruction between psp and nsp of the entire expression. That got me rid of the infinit loop, but I didn't get the right order.

I also can't figure out how to express the above 14 steps in the SSA form so that the DEST register is assigned to only once and there exists a single pseudo instruction which can be denoted as the origin of the register value.

Well, I guess I need to keep on trying. Every failure like this teaches me something new from Lestes and maybe it's worth the trouble just because of that.

Report from the AMD64 mini CDA training at MFF UK

2008-05-07T10:48:00.016+02:00

In my previous post, I announced the AMD64 crash dump analysis course that me, Vita Batrla and Vlada Marek delivered at the Faculty of Mathematics and Physics, Charles University in Prague (MFF UK) last week. From our point of view, the three-day event was a success. We explained the AMD64 stack layout in detail, taught how to use mdb to analyze core dumps and kernel crash dumps, and let the trainees work on pre-prepared crash dumps showing different kind of kernel problems. In particular, we've looked at crashes caused by some garbage pointer value stored in a kernel variable, problems with the AMD64 stack red-zone, memory corruption bugs, deadlocks and hangs. We had 23 people registered, out of which 20 really showed up. Those people were mostly from MFF UK, but there were also people from other universities, namely FEL CVUT and FJFI, as well as people from Czech IT companies.

Mini AMD64 CDA training at MFF UK

2008-04-23T14:04:00.003+02:00

I am excited about the three-day course that I and two colleagues of mine are going to deliver at the Faculty of Mathematics and Physics, Charles University in Prague, next week. The course will be dedicated to crash dump analysis of amd64 OpenSolaris kernels. If you speak Czech, you can check out the invitation here.

Show me your ID!

2008-04-17T03:39:00.008+02:00

I am currently to be found in Sunnyvale, California, USA. The weather and people here are warm and I am generally enjoying my stay. What I especially appreciate, and I am becoming a little sarcastic now, is that in 2/4 attempts to buy beer I was asked to show ID. Being European, I obviously don't have the Californian ID, so I would always show my Czech/European passport instead. The first guy who had not believed me being 27 was satisfied when he saw my passport and kindly explained why was it necessary that me and other two of my colleagues, aged around 30, had to show ID as well. Today, while I was shopping in a supermarket, I bought myself two bottles of European beer - (Guinness and Heineken; had they had Czech beer, I would have gone with that instead). The cashier was very very reluctant to accept my passport as a proof of my entitlement to buy alcohol in US so I was beginning to despair - I've been a little thirsty in the last couple of evenings. Fortunately, the validity of my passport was eventually ACK'ed by the cashier's boss and my alcohol purchase could proceed.

The good news is that I must look much younger than I actually am. Even though I am married and I already have traces of silver in my hair, people still think I am no older than 20. Maybe the years of sitting behind a computer have preserved me for future generations of US cashiers :-)

Coreboot project uses bits of HelenOS

2008-03-21T10:16:00.002+01:00

I've just found out that the coreboot project (formerly known as LinuxBIOS), more precisely its libpayload, contains small portions of HelenOS libc. So far, they've taken our memory string operations and also the whole printf subsystem. This is a good sign that other projects can, and actually do, pick up parts of HelenOS and modify them for their own needs. Hopefully, this trend will continue.

Dangerous Direction Flag

2008-03-20T23:40:00.004+01:00

In a news message on root.cz, someone has confused the newly introduced GCC 4.3 optimization of not including the extra CLD isntruction in front of each string instruction with a GCC bug. The title of the news read:

GCC 4.3 contains a bug

After reading the news, I figured out that it is not GCC which contains a bug, but the Linux kernel. I am not going to talk about this bug in this post (read more about it here). What I will talk about is, however, very related.

Learning more about the Linux bug made me realize how dangerous the seemingly innocent DF flag can be to the kernel. Without preventive measures, the reversed direction of string operations could cause severe kernel memory corruption. I also realized that we don't probably take those preventive measures (i.e. clearing DF upon entering the kernel) in HelenOS/ia32 and HelenOS/amd64. Quick check of the respective sources confirmed my worries.

Later at home, I did basicaly two things. The first was that I modified HelenOS libc to always do STD before doing a syscall and verified that the problem was real. The second thing was implementing the fix for this HelenOS bug.

On ia32, I inserted one CLD into the interrupt handler macro and one CLD into the syscall handler macro and that was it. On amd64, I used the CLD instruction for the interrupt handler macro as well, but wanted to make use of the amd64's SFMASK MSR for the syscall path instead of the boring CLD. The SFMASK register contains a mask that will be applied to the RFLAGS register during the SYSCALL instruction. In other words, a man can program it to clear arbitrary RFLAGS bits. When I was done with amd64, I wanted to test the fix in QEMU. To my surprise, fixed HelenOS/amd64 didn't function well in QEMU. I repeated the same test with Simics, where everything worked fine. Hm, a bug in QEMU? Looks like it (and today I found out what was wrong).

To top the DF story off, I have to confess that for some (rather short) time I believed that there was a similar bug in amd64 version of Solaris. Contrary to the ia32 version, there is no CLD at the beginning of the respective interrupt handler (i.e. cmntrap) and the syscall handler. In case of the syscall handler I noticed that the SFMASK MSR is only programed to clear the IF and TF, but not DF. Eager to find a bug, I even started to build a crafted Solaris libc, that would demonstrate the bug as I did with HelenOS. But then I finally got it: Solaris brings RFLAGS into a known state by pushing a pre-defined RFLAGS value onto the stack followed by the POPF instruction. I was so focused on the CLD which wasn't there and the insufficient SFMASK that I didn't see what was there all the time.

Lestes wishlist

2008-02-06T22:54:00.000+01:00

This is an incomplete list of things I would like to see in the Lestes project, sorted in the order of achievability (i.e. as I can achieve them):

integration of my GAS assembler generator patches
retarget the compiler to mips32
remove unnecessary frontend dependencies on backend from the frontend
make use of the branch delay slots for useful instructions
support for 64-bit long long type
retarget the compiler to amd64
full support for ia32's and amd64's disp(base, index, scale) addressing in the generated code
fix semantic errors (e.g. long long long long type)
implement the missing logical operators (i.e. || and &&)
implementation of the 'cee' frontend
retarget the compiler to other processors, especially those supported by HelenOS

File system support for HelenOS

2008-02-03T17:28:00.000+01:00

I feel we are close to a new HelenOS release which will bring basic file system support. So what has been brewing in the repository for some time already? We basically have two or three components that will be ready for a review by releasing it. Namely, we have VFS, which is the place where all information about registered file systems and open files live. VFS also provides a single interface to the rest of the world through which it is possible to communicate with the underlying file system implementations. We also have a prototype implementation, TMPFS, which is a file system that lives completely in virtual memory and has no disk layout. I have also started FAT support, but it is now resting in favor of work that needs to be done on TMPFS. Finally, we have DEVMAP, which will register and provide connection services to registered block devices.

So what is already working? We can do simple mounts of TMPFS as a root file system. We can look up files and directories on TMPFS. We also can open(), read(), write(), lseek() and ftrunc() files and opendir(), readdir(), rewinddir(), closedir() and mkdir() directories.

It's already better to say what functionality is missing. We don't yet support rmdir(), rename(), unlink() and close().

You may say so what, TMPFS will lose data on next reboot, it's like having no file system support at all. The reason of doing TMPFS first is, of course, developing VFS and libfs frameworks so that other file systems can be added faster and with more ease. It will also help to blunt the sharp edges of the current implementation.

Fixing tmpfs in Solaris

2008-02-03T15:50:00.000+01:00

A year ago, I started to work on a problem one of our customers had with the tmpfs file system. Since the problem has been already fixed in all relevant Solaris and OpenSolaris releases, I feel I can share some of the more interesting technical bits.

The problem, as laid by the customer, was that the system would hang if an attempt to fill a tmpfs filesystem (e.g. /tmp) is made. In other words, a mere:

dd if=/dev/zero of=/tmp/testfile

would slowly hang the system.

During the course of investigation, we actually found two bugs that contributed to this hang, even though the system would be usually only extremely slow and non-responsive, but not completely hung.

The first bug showed when one process held the majority of all available memory pages in the dirty state and wanted to allocate yet another page for itself. When there were no more available pages, this process got blocked waiting for the pageout thread to pageout the dirty pages to swap and make them available again. The pageout daemon walks the pages and when it finds a dirty page, it uses the respective vnode's putpage routine to do the actual pageout. In case of tmpfs, the putpage routine has a check that verifies that the tn_contents rwlock (i.e. the lock protecting the contents of the tmpnode to which the dirty page belongs) is not being held. If it is, putpage simply gives up this page and moves on to another one. Now, the problem was that when the dd process went to sleep, it held the tn_contents lock of /tmp/testfile from our example command. Moreover, almost every single page in the system belonged to this file and was dirty. The result? The pageout daemon could not do any forward progress as it had to give up paging out majority of pages due to the held tn_contents lock and the dd process could not unlock the tn_contents lock because it wanted another page.

This bug got fixed by modifying the wrtmp() routine, which is on the write(2) execution call path. The fix simply dropped the tn_contents lock before the thread in wrtmp() would get blocked waiting for a page and reacquired it later. Nevertheless, the problem didn't go away completely and we learned we only cured half of it (for the side effects of this cure, read on), but maybe the more serious half.

It turned out that a fix for large ISM segments on systems with ZFS introduced a change in the reservation of anonymous memory which was a regression for tmpfs and the second half of the required solution. After the addition of ZFS , databases which needed to create large shared mappings started to fail (i.e. were unable to create these large shared mappings) due to ZFS caching memory, which would otherwise be necessary for the shared segment. This was fixed by inserting a call that reaped several different caches in front of the test in anon_resvmem() that checks the amount of reservable memory (i.e. availrmem) and fails if the anon_resvmem() request cannot be satisfied. The problem with this is that this fix made it actually much harder for a tmpfs allocation to fail. Additionally, the procedure which reaped the caches was pretty heavyweight and contained a loop, which could delay every anon_resvmem() by as much as by 60 seconds! What? I said the theoretical maximum delay per request was 60 seconds. In the lab, I was able to reproduce this and using a clever dtrace script, I measured the maximum delay of 13 seconds, which is still pretty horrible.

I fixed this by selectively disabling the cache-reap for tmpfs completely so that now when there is no reservable memory, the dd command above will fail. Users can still guarantee tmpfs reservations by creating disk swap of sufficient size. Everything which is beyond the disk swap size, is not guaranteed and the reservation may fail. Note that this behaviour is consistent with the documentation for swap.

As it turned out, the fix for the latter bug was sufficient because it prevented the former bug from occurring. But it was difficult to see the latter bug without first fixing the former one. Moreover, the customer didn't make use of the possibility to crop the tmpfs size by the "size" mount option, which would have also solved the problem.

Finally, let me go back to the side effects of dropping the tn_contents lock. Due to the implementation of wrtmp(), the act of increasing the file's size and creating the new portion of its content was no longer atomic as seen from the perspective of a process which writes to a tmpfs file and mmaps its end and tries to read it at a time. Although documented and forbidden, such a behaviour was a nuisance. So I slightly reordered wrtmp() and putback the fix for this last Tuesday.

As of now, I am still not completely done with tmpfs and will come back to this topic later, when there is a little more to add.

New assembler generator for Lestes

2008-02-03T14:42:00.000+01:00

During the past week, I've been playing with the Lestes compiler again. I found it pretty non-satisfactory that the compiler could generate .asm files only in the nasm syntax. I decided to try to add the ability to generate gas syntax as well.

Besides nasm and gas are different assemblers, the former generates code in what is called Intel syntax, while the latter uses AT&T syntax. The main difference between the two is the swapped order of instruction operands and a different way of specifying memory operand sizes.

Fortunately, this adventure took me only several days and editing a couple of XML files which describe the target machine. Besides of that, I also had to fix several dependencies on nasm which were hardcoded directly in the compiler's source code. My fixes are now ready for code review and will hopefully be integrated into the project's mainline repository sometime soon. Looks like this experience will be useful in my future Lestes endeavours, especially if I decide to retarget the compiler to amd64 or any other platform. BTW, nasm doesn't support anything but ia32 and amd64, so if Lestes can generate code for sparc64 one day, it will have to be assembled by gas and not nasm.

So how do you actually make Lestes generate gas assembler? It's just a matter of replacing the machine description XML files with those that are meant for gas. The following should do:

$ cd target/machine/i686/tm_description/
$ build_md.sh gas
$ cd ../../../..
$ make