Friday, September 9, 2016

HelenOS Camp 2016 Report

By tomorrow it will have been a week since we decamped from HelenOS Camp 2016. So while the memory is fresh, it's time for a short summary.

The traditional showcasing of gadgets of all sorts near the beginning of the camp

The camp took place some 100 kilometers east of Prague in a rented cottage, virtually a stone's throw away from the Lichnice castle. The total of 45 changesets went into the mainline during the camp and some more are waiting in the pipeline. Here is a breakdown of the camp development activities per attendee:

The Lichnice castle offered some fine panoramas

Jakub (JJ) worked on:

  • generic user pager framework and a simple pager implementation in the VFS server capable of private file mappings; shared mappings are not supported yet because more work is needed: a cache for tracking pages served by the pager and a software solution compensating for the missing dirty bit in the hardware PTEs on arm32 and ppc32;
  • modification of the page mapping interfaces so that page_mapping_find() returns a copy of the actual PTE and a new interface called page_mapping_update() for updating the actual PTE dirty and accessed bits from a copy; this allows the global page hash table lock (on ia64 and sparc64) to protect much smaller critical sections and avoid deadlocks with the user pager code;
  • removal of sparc32 support due to the lack of maintenance and suitable debugging/development tools; this is not necessarily the end of story for sparc32 - if the situation improves wrt. tooling, I am fond of reviving it;
  • still-unfinished upgrade to QEMU 2.7.0 held up by an Integrator/CP regression/oddity.

Martin (MD) reports:
Before the camp begun I set myself the primary goal to finally finish the RISC-V support in HelenOS. Unfortunately, fulfilling this goal has turned to be less straightforward than originally expected, mostly due to the changes in the draft of the privileged specification of RISC-V and even more importantly due to the changes in the Spike reference simulator of RISC-V since late spring 2016. Because both the privileged specification and the simulator are still not considered frozen/stable, some of the changes are not explicitly tracked and documented, thus making the debugging process slightly tedious at times. Due to these unexpected obstacles I have only managed to commit a dummy placeholder implementation of the RISC-V support into the HelenOS mainline branch during the camp. This placeholder implementation can be successfully built using the RISC-V specific cross-compiler toolchain, but it is not actually operational.

Since the work on the RISC-V has been somewhat less rewarding at times, I have also reviewed the commits to the HelenOS mainline branch from the recent past and made some cosmetic fixes to the code -- mostly coding style improvements. I have also implemented a very trivial GUI calculator (inspired by an independent student project that appeared on GitHub some years ago) as a demonstrator for adding the support for keeping user data in the GUI widgets. Finally, I have implemented an optional support for running the HelenOS kernel in the 'large' memory model on AMD64 for the purposes of functional comparison with RISC-V.

Jiří's (JS) account of the camp:
I spent one day trying to debug why my TP-Link (Atheros) WiFi is not working, but without much success. The driver simply fails to receive a response from the device at one point. I spent the rest of the time implementing an ordered dictionary data type based on red-black trees. It is roughly comparable to the B+tree currently available in the kernel, but has a few additional advantages (e.g. all memory is pre-allocated by the caller). I also implemented basic unit tests which helped me verify basic operation. There is still a little bit of work left before integrating, including one known bug that I've discovered after the camp and some optimizations.

Stanislav, a CS student working on porting the BIRD Internet Routing Daemon to HelenOS as his thesis, paid us a one-day visit to demo and also work a little bit on, well, just that: porting BIRD to HelenOS.

Apart from coding, of which there was plenty (12-hour shifts not being an exception), we managed to make a short trip to the Lichnice castle, have two BBQ's, stargaze and drink beer.

BBQ with current and future developers

All in all, we enjoyed a fun and productive week with a lot of opportunities to talk and plan ahead. I am already looking forward to HelenOS Camp 2017!

Tuesday, January 19, 2016

Running HelenOS/sparc64 in gem5

On Sunday I created a simple shell script to install and configure the gem5 simulator for use with HelenOS (and other operating systems). The script closely follows, streamlines and automates most of the steps described in this blog written by Gedare Bloom. In particular, it takes care of:
  • cloning the gem5 repository;
  • building the gem5.fast binary for SPARC;
  • downloading and extracting the firmware blobs from the OpenSPARC tarball; 
  • creating symbolic links to the .bin files so that gem5 can find them under the name it expects;
  • copying out the gem5 configuration files and patching example/fs.py to propagate the --disk-image script option to the actual configuration (this should really be the default, IMO).
When the script runs to completion without an error, one can start the simulation by running the following command line from the directory in which the script was executed:
M5_PATH=`pwd` \
gem5/build/SPARC/gem5.fast configs/example/fs.py \
--disk-image=/path/to/image.iso
After doing this, gem5 should output a couple of lines similar to these:
gem5 Simulator System. http://gem5.org
gem5 is copyrighted software; use the --copyright option for details.
gem5 compiled Jan 17 2016 23:03:08gem5 started Jan 19 2016 11:25:35
gem5 executing on gorgo, pid 11963
command line: gem5/build/SPARC/gem5.fast configs/example/fs.py --disk-image=/home/jermar/software/HelenOS.mainline/image.iso
Global frequency set at 1000000000000 ticks per second
warn: DRAM device capacity (8192 Mbytes) does not match the address range assigned (64 Mbytes)
warn: DRAM device capacity (8192 Mbytes) does not match the address range assigned (256 Mbytes)
info: No kernel set for full system simulation. Assuming you know what you're doing
Listening for t1000 connection on port 3456
Listening for t1000 connection on port 3457
0: system.remote_gdb.listener: listening for remote gdb on port 7000
**** REAL SIMULATION ****
info: Entering event queue @ 0. Starting simulation...
info: Ignoring write to SPARC ERROR regsiter
info: Ignoring write to SPARC ERROR regsiter
warn: Don't know what interrupt to clear for console.
From another terminal one can connect to the serial console either via:
telnet localhost 3457
or:
gem5/util/term/m5term 3457
After a short while, the simulated machine should output this to the serial console:
==== m5 slave terminal: Terminal 0 ==== cpu Probing I/O buses

Sun Fire T2000, No KeyboardCopyright 2005 Sun Microsystems, Inc.  All rights reserved.OpenBoot 4.20.0, 256 MB memory available, Serial #1122867.[mo23723 obp4.20.0 #0]Ethernet address 0:80:3:de:ad:3, Host ID: 80112233.


ok 
I am planning to integrate the gem5 support into the tools/ew.py script in order to further automate the process of starting the simulation of HelenOS/sparc64/sun4v. I will also try to figure out how to get rid of the nuisance of having to type boot into the ok prompt at the beginning of each simulation. Theoretically, it should be possible to do:
ok setenv auto-boot? true
and then somehow save the contents of the NVRAM into the nvram1 binary as installed by the script.

Unfortunately, gem5 is not yet quite ready for running mainline HelenOS because it has problems with the TICK and TICK_COMPARE registers. There has been an incomplete attempt to fix this, but for the time being HelenOS needs to be patched to use STICK and STICK_COMPARE for better simulation experience.

Even after working around the TICK_COMPARE issue, HelenOS experience in gem5 will not be smooth enough because of two unresolved issues. The first issue is that for some reason the VFS server crashes when the TSB support is enabled. The TSB can be thought of as a large shared software-controlled TLB and is not essential for running HelenOS, so configuring HelenOS not to use TSB will help the simulation a lot. The remaining second issue is that sometimes the STICK register will get ahead of the STICK_COMPARE without generating the timer interrupt, which will effectively render HelenOS wedged.

I am looking into these issues as time permits and will report in the comments section or in a new blog post when there is some progress to report.

Wednesday, September 30, 2015

Cache management overview across ARM architectures

I've recently run into troubles caused by subtle differences in how the different ARM subarchitectures or their implementations manage caches. Sometimes an implementation does not implement what its respective subarchitecture mandates (or rather suggests), sometimes it implements more. ARMv4 to ARMv6 are quite similar, but ARMv7-A is completely different, because of the possibility to have multiple levels of caches across multiple processors and the new concepts of points of coherence and unification that come with it. When dealing with these differences, I wished there was a simple table that would tell me what subarchitecture supports what features, but there wasn't... Until now. The following chart depicts the situation for ARM processors and subarchitectures supported by HelenOS.

ARM cache management via CP15 register C7

Monday, March 9, 2015

Pandora, Penelope and Sisyphus

So I read Mr. Duff's latest book Pandora, Penelope, Polity: How to Change the European Union in a hope to find a well-reasoned answer to my curious question: "Why the author thinks that federal Europe can be achieved by a treaty change." My findings below might be of sensitive nature to some, but I consider it necessary as writing down what one really thinks is the only way to have a reasonable debate, or actually any debate at all, about federalism among those who like to use that word. The first action in the attempt to unite the federalists should be to agree on the basic terms and definitions, otherwise we will get forever lost in the Babylonian maze of speculations of what the words of others could possibly mean. Reaching that agreement should not be that difficult though, because the definition of a federation is not an open-ended question and the answer to it is not arbitrary. The requisite features of federal systems have been identified and described by several distinguished political scientists in several writings.

Sisyphus (1548–49) by Titian
The book itself does a pretty good job in explaining the complexities of the institutional system of the European Union, including several attempts - past and suggested - to improve it by a treaty change. Unfortunately, the book does not say why we should stick with the system of treaties in the first place and not aim for something better, it only keeps on repeating the mantra that a treaty change is necessary. It simply puts it as a given fact and does not consider, let alone mention other possibilities with successful precedents in the past, like having a classical bottom-up federation founded by the citizens. For a book about the federal future of the European Union, this omission or avoidance must be seen as a major issue.

According to the author, the outcome of the next treaty change should be a democratic federation, composed of states and citizens. But paradoxically this Duffian federation has the form of an international treaty, is founded by the states and operates mostly on the states. States can unilaterally secede from it, so in fact it is not a federation at all and strongly contrasts with the idea of a federal republic created by the people. With all respect, I have no idea why Mr. Duff associates the adjective federal with it. If someone knows better and spots an error on my side, I'd love to be educated by them in this matter. If I may, for the time being, I'd also like to suggest a more fitting adjective that would have raised no eyebrows: confederal.

If one takes an intergovernmental treaty and applies some heavy lifting, streamlining, renaming and tinkering to it, but does not change the confederal substance, he will not arrive at a federation, but at a more integrated intergovernmental treaty. Calling things federal when they aren't does not make them any more federal. It's like if Mr. Duff could not liberate himself in his writing from the confines of what he knows best: the institutional order of the present European Union.

Proponents of this idea are therefore sometimes reminiscent of little industrious creatures that live in a one-dimensional world, honest in their intentions and sincere in their efforts, but nevertheless completely ignorant of the other dimensions. They can move in either direction on the axis also known as the European integration, so they move forward, hoping to find the European political sweetspot in front of them. Interestingly enough, the Eurosceptics are the same kind of creatures, only moving in the opposite direction. Unfortunately, neither of them is able to find the sweetspot, because it is not situated on the axis. In order to reach that point, they need to become aware of the other dimensions and take a bold, discrete step aside. A step towards thinking from the citizens and a real federal organization.

Tuesday, February 10, 2015

Thoughts on The Anti-Federalist Papers and the Constitutional Convention Debates by Ralph Ketcham


The Anti-Federalist Papers and the Constitutional Convention Debates is a very interesting book that will draw the reader right into the debates of the Constitutional Convention in Philadelphia in its first half, and into the minds of the anti-federalists in its second half.

While it was quite interesting to be like a fly on the wall of the Independence Hall in Philadelphia, listening to the recorded words of the Convention delegates, the book remains silent on some of the most interesting moments, such as how exactly was the Great Compromise reached and how they abandoned the idea of the negative of the general government on the state laws. The book also reveals two strange facts or rather curiosities about two of the delegates.

Edmund Randolph, then governor of Virginia, was the man who officially proposed Madison's Virginia plan to the Convention, but then was one of the three delegates who refused to put their names under the finished draft Constitution, which was derived by (a weaker) compromise from that very same plan. One year later, during the Virginia ratification convention, he changed his mind again and voted for ratification.

Alexander Hamilton proposed his own plan for national government, but nobody cared. On at least two occasions he publicly said he disliked the proposed Constitution. I think no one could have suspected him of being Publius. Was that a camouflage?

The arguments of the anti-federalists were also interesting, for example that the President together with the Senate could make any law through his power to make treaties with foreign countries that in turn become the supreme law of the land, encroaching thus on state laws. Other than that, I noticed one large group of the anti-federalist arguments was effectively neutralized by the adoption of the Bill of Rights (a concession of the federalists to the anti-federalists or a very clever move). Another large group consisted of, from today's point of view, unsubstantiated worries of the anti-federalist and absurd fears. For example, they were pretty sure that the Constitution could not be amended (today it has 27 amendments). Or they feared that the number of Representatives can be pushed down to one per each state. Or that the state governments will be abolished. Or that the words general welfare in the Preamble will allow the federal government to tax the people to death. I felt like experiencing a déjà vu when I was reading those parts. Anyway, the history has proved them wrong.

It was also interesting to feel the atmosphere of the debates. Much more cultivated than what we have today in Europe. The anti-federalists advocated for caution and deeper understanding of the proposed Constitution before its adoption, or proposed amendments. Far from today's heated internet discussions with shallow knowledge, superficial understanding and tunnel vision.

Friday, October 31, 2014

Highlights of A Concise History of Switzerland

I have finally finished reading A Concise History of Switzerland from the Cambridge Concise Histories series by Clive H. Church and Randolph C. Head. As I was reading it, I underlined a couple of paragraphs that contain the most interesting parts (at least for me). I am now going to paste these highlights here and attach a short commentary to some of  them. Needless to say, it's up to you to make your own opinion. Maybe you will want to read the book in its entirety in a desire to reflect on the possibility to learn from someone else's fails and wins a little more.

Neither dynasty, nor language, nor religion brought about a Swiss national identity that could bolster a Swiss political nation. Instead, modern Switzerland seems in an important sense the result of its inhabitants’ own decisions – a Willensnation, a nation resting on its inhabitants’ will – and of its own and its neighbours’ willingness to accept its various forms through the centuries as a single and continuous political unit.
This is my first highlight and it conveys perhaps the strongest message of the whole book. It's like an affirmative follow-up on the question mentioned in the third sentence of the opening paragraph of Hamilton's Federalist No. 1, which goes like this:

It has been frequently remarked, that it seems to have been reserved to the people of this country, by their conduct and example, to decide the important question, whether societies of men are really capable or not, of establishing good government from reflection and choice, or whether they are forever destined to depend, for their political constitutions, on accident and force.
Going back to the book, some of the following highlights (from all over the book) support the idea of a Willensnation and debunk the homogenous people and language fallacy:

Although all of the original eight cantons spoke German, and shared a certain contempt for Romance, especially Italian, mores and attitudes, their shared language did not in itself increase their cohesion or limit their alliances with neighbouring Romance-speaking communities or lords. As in the polyglot kingdoms around them, language played only a minimal role in Swiss nation-formation before the seventeenth century.
---
Despite the hazards, many Swiss intellectuals were drawn towards thinking about their own Swiss identity and asking what held them together despite their linguistic and religious differences. The late 1750s saw the publication of a number of introspective writings, such the Patriotic Dreams of both Franz Urs Balthasar from Lucerne and Isaak Iselin from Basle. The latter was historian, philosopher and critic both of Rousseau and of Switzerland’s mercantile and aristocratic spirit. Balthasar and others were even more critical of the state of the Confederacy, the former proposing a federal college, a unified army and a confederal tax to remedy its weaknesses. More than in previous centuries, intellectuals now thought of Switzerland as a single nation, though one sorely lacking in political unity. Folk songs and the increasing interest in the mountains helped to further the idea of distinct Swiss and Alpine identity.
--- 
The leaders of the new state were able to forge a new nation based on political will and the key institutions of direct democracy, federalism and neutrality.
The last quoted paragraph above refers to the period after 1848 when Switzerland finally became a classical federation operating on the terms of a federal constitution. However, in the following chronological sequence of highlights, I would like to draw your attention to the deficiencies of the several old Swiss political systems before 1848 - something which will seem igloriously familiar to most contemporary Europeans:
The Diet provided a site for discussions of such problems, but negotiation also took place before the councils of individual cantons, or in secret with leading political figures.
---
Persistent resentments helped, in other words, to prevent any rapid change to the ramshackle constitutional structures of the Confederacy and its subject territories. It remained a contractual system based on a network of treaties, respect for the established order and multiple layers of privilege. Hence the Diet was little more than a symbol and a means of socialization, lacking army, finance and a proper administration as it did. Indeed, if anything, the Diet became even less active at this time because of the need to meet in private and refer all decisions back to the cantons. The joke circulated that the Diet would not agree that snow fell in winter without asking for instructions. Even when decisions were finally reached, cantons were not obliged to act on them.
---
The Confederacy thus remained a weak body, lacking a constitution, an army and a reliable tax-based income . Far from being a state comparable to its more powerful reforming neighbours, it remained self-satisifiedly dependent on an implicit French guarantee, which in turn depended on a balance of power that had turned against France following its defeats in the Seven Years War. Hence the suggestion made by Johann Georg Stokar, a chairman of the Helvetic Society , that the country needed to unify and accept equal rights for all, was ignored. Indeed, things were already going in the opposite direction.
---
Refusing ideas of unification was, in other words, symptomatic of the oligarchies’ approach in the 1770s and 1780s. Going beyond immediate challenges, the oligarchs sought aggressively to entrench their positions and interests at the expense of lesser breeds, something that prompted very significant risings in Geneva and Fribourg.
---
Such victories won the patriciates a few more years of unchallenged peace, but these were not used to institute reforms that could have helped them cope with rapidly changing circumstances at home and abroad. The authoritarian oligarchs continued to see no need to adapt. They preferred to rely for their legitimacy on custom, heredity and closeness to an established church. Indeed, they saw themselves as a divinely ordained elite at the peak of a hierarchical society. In this they eschewed the newer representative ideas emerging in the Anglo-American world in favour of continental traditions of the seventeenth century.
---
At the confederal level, the inherent weakness of the system became all too apparent after Schwyz in 1789 refused to agree to a renewal of the Defensionale, which coordinated joint efforts for defence of the borders. Likewise, the St Gallen politician Müller-Friedberg’s call for a new charter between all the cantons and allies had no effect. Such continuing preference for local over general interests made it almost impossible for the Confederacy to muster sufficient political and military force to resist French encroachments. It remained an assemblage of self-interested rural and urban communities. As a result, the Diet failed to agree on any emphatic measures until very late in the day, because disagreements and traditional arrogance continued to dominate, even though passive stability was no longer enough.
---
Nonetheless, in all cases it was the cantons that provided sovereignty . The fact that the confederal constitution came at the end was symptomatic of this. The new regime was chaired by a rotating directing canton, known as the Vorort, whose head was the Landammann of the new executive, a body with rather limited powers.
---
The new Charter was essentially a treaty amongst sovereign cantonal states. The cantons mutually guaranteed their own security and constitutions – often quite backward-looking – and sent representatives to the Diet under strict mandates. The Charter did not provide for a Landammann, a national judiciary or separation of powers. Nonetheless, it represented a step beyond the Mediation and towards a Swiss Confederation, the term being formally used for the first time. It contained a commitment to a common fatherland and gave the Confederation new powers. Significantly, the Diet could make binding decisions by a two-thirds majority. Cantons were banned from creating leagues amongst themselves, and the Diet gained responsibility for upholding confederal decisions, a new power, as well as for trade, diplomacy and defence. For the first time, a national army was established , even if it was made up of cantonal contingents.

Friday, December 6, 2013

Where we stand in the world

Here is a map of the world depicting the 15 non-EU members of G20 (red) and the 28 EU states (green).

G20 and EU
According to the g20.org site, the G20 represents about 85% of global GDP, over 75% of global trade and 66% of the world's population. Confronting this with a Wikipedia page on G20, one finds out that the percentage for the global trade is slightly off, because it includes Germany, France, UK and Italy numbers twice - once for these countries themselves and once for their EU membership. Thus, the correct number seems to be rather 58% of global trade for G20.

Now, if we take the EU completely out, just to see against what economic and geopolitical power Europe competes, we get something which could be called a G15 (G15 = G20 - Germany - France - UK - Italy - EU). Note that there seems to be an informal group of countries of the same name, so do not confuse it with our G15.

G15, even after the removal of the entire EU, is still a major group, representing 63% of global GDP, 45% of global trade and 57% of world's population. As a side note, the majority (8) of these countries are federal states (also large and multiethnic) or at least declare themselves as such. The smallest G15 member, Australia has population of only 22 million. The middle-sized member, Mexico, has 112 million inhabitants and the two largest countries in this group, China and India, have 1.2 and 1.3 billion people living in them, respectively.

When we put the EU into the same perspective (these are all figures from 2012, so Croatia is not included, sorry), we find that it is responsible for 23% of global GDP, 12% of global world trade and has 7% of the world's population. Note that this is not even the whole of Europe, just EU-27.

Let us make one additional daring step and have a look at how e.g. UK, a country that sometimes likes to picture itself as an independent state outside of the EU, performs in this global game. UK, as an independent country, creates around 3.4% of global GDP, 3% of global world trade and has about 0.89% of the world's population. Just draw your own conclusion, considering also the dynamics of the UK and G15 economies and populations.

After having opened this blog entry with a picture, would there be a better way of concluding it other than using another picture to illustrate the dramatic change through which Europe's position in the world has gone since 1945? Here is a map showing, among other things, colonies of several European countries in 1945. Colonies that are no longer there and will never come back.

Colonialism in 1945, source: Wikipedia

G20 members represent around 85 per cent of global gross domestic product, over 75 per cent of global trade, and two-thirds of the world’s population. - See more at: http://www.g20.org/about_g20/g20_members#sthash.rXrn8w8i.dpuf
G20 members represent around 85 per cent of global gross domestic product, over 75 per cent of global trade, and two-thirds of the world’s population. - See more at: http://www.g20.org/about_g20/g20_members#sthash.rXrn8w8i.dpuf
G20 members represent around 85 per cent of global gross domestic product, over 75 per cent of global trade, and two-thirds of the world’s population. - See more at: http://www.g20.org/about_g20/g20_members#sthash.rXrn8w8i.dpuf
G20 members represent around 85 per cent of global gross domestic product, over 75 per cent of global trade, and two-thirds of the world’s population. - See more at: http://www.g20.org/about_g20/g20_members#sthash.rXrn8w8i.dpuf
G20 members represent around 85 per cent of global gross domestic product, over 75 per cent of global trade, and two-thirds of the world’s population. - See more at: http://www.g20.org/about_g20/g20_members#sthash.rXrn8w8i.dpuf
G20 members represent around 85 per cent of global gross domestic product, over 75 per cent of global trade, and two-thirds of the world’s population. - See more at: http://www.g20.org/about_g20/g20_members#sthash.rXrn8w8i.dpuf
G20 members represent around 85 per cent of global gross domestic product, over 75 per cent of global trade, and two-thirds of the world’s population. - See more at: http://www.g20.org/about_g20/g20_members#sthash.rXrn8w8i.dpuf