Why Ubuntu Should Continue with Upstart for 11.10

So I just read Lennart Poettering’s “fair and balanced” review of sysvinit, upstart, and systemd….wow.  Looking at his comparison chart, we in Ubuntu must be idiots to not switch over to systemd immediately…especially since he clearly points out all the major distributions have done so (or plan to) already.   Once again, the evil Mark Shuttleworth must be dictating that Ubuntu must remain on upstart, oppressively pushing down all those who challenge his rule….whatever people.  So here’s the real reason why I think we should remain on upstart in 11.10, it’s because (as Mark mentioned todaywe put users first.  Do I need to remind anyone of the pain we went through in Karmic (Ubuntu 9.10) when we finally made the wholesale jump to upstart?  Sure, we achieved great boot performance gains, but it was painful, especially for Ubuntu Server, as it was largely neglected during that effort (and I’ll take the blame for that).  We spent the next release, Lucid, cleaning up behind ourselves….frantically working to get the next LTS in a respectable shape…and still, Ubuntu Server was neglected (again, blame me).

So here we are again, one release before an LTS…an LTS that is not only going to showcase the quality of the Desktop, but is going to be extremely important for Ubuntu Server, and people are asking us to switch to systemd?  Really??  We just got done improving upstart, making upstart play nicer with Server, rolled out a damn nice user guide, and even added some slick features (like  job and event visualization)…and we’re supposed to throw all that out and switch to systemd now?   The situation reminds me of a quote from one of the funniest (and probably worst) presidents in US history:

“There’s an old saying in Tennessee — I know it’s in Texas, probably in Tennessee — that says, fool me once, shame on — shame on you. Fool me — you can’t get fooled again.” -George Bush

But seriously (with all joking aside), I don’t want to go through a rushed change again, which is why I support staying on upstart for both 11.10 and 12.04 LTS, and then taking a serious look at the merits and drawbacks of moving to systemd going into the 12.10 cycle.  Basing a decision on what we feel is important to Ubuntu and it’s users, not Lennart.  By then, systemd will have another year to mature, we don’t have an impeding LTS release on our backs, and if Debian is to truly switch to systemd, then a year’s wait while that work goes on should only improve the chances of Ubuntu adopting it.

Lennart, if you by chance read this, can you please stop the campaign and badgering against us…it ultimately does you no good.  We aren’t pushing back because we don’t like you, or Fedora, or because Mark is forcing us to stay with upstart…it’s because we put users first.  While I agree upstart isn’t perfect, and certainly still causes server sysadmins pain in some situations, I’d rather deal with the problems we have with it , than take a leap of faith with systemd this close to an LTS.


About Robbie

I live in the awesome city of Austin, TX and work for Canonical, sponsors of the best damn operating system in the world...Ubuntu.

Posted on April 29, 2011, in Ubuntu and tagged , . Bookmark the permalink. 37 Comments.

  1. Understandable decision. You guys know upstart and its quirks. Staying with it and letting Debian do the transition first and then profit from their work is probably the sanest way to go.

  2. It sounds reasonable to me. Sticking with upstart is probably a good idea, at least until next LTS.

    But man… stop the “defensive whining”, because it add some noise to the well reasoned part of your post.

    • Sorry you see it as “defensive whining”…maybe you have a point…but I’m just so tired of all the baseless and unsubstantiated attacks we’ve taken over this.

  3. Maybe you should not blame yourself, but upstart?

    I have experience with both upstart and systemd on servers. While I never felt comfortable with upstart on them, systemd felt like it belonged there from the first minute on. In fact, the day after migrating to systemd on the first server, I already started feeling uncomfortable with classical sysv on all others.

    • Well, being that I managed the Foundations team and supported (still do) the decision to move to upstart…I deserve some blame for not making sure we paid closer attention to the affects on Server.

  4. The post is fine other than your statement about putting users first. If that were true you’d give them something great and stable. Currently you are actually only giving them something stable. This isn’t bad, not at all, but state it for what it is.

  5. I haven’t tried systemd yet, but it seems way better than anything else. Perhaps it’s time to shift some of Canonical’s effort from upstart to systemd.

    Converting just before LTS would be a bad idea. But there’s one full cycle before that! Systemd should be considered for 11.10. I’d love to see it there.

    I’ve read both posts. Your post is so much more offending! Please keep the arguments and decision technical, not personal! This post only makes it worse 😦 Shame. Does it have to be about developers EGOs? It’s USERS first, isn’t it?

  6. I also wish Lennart would remember what happened (and is to some degree still happening) with pulseaudio. That was a long, and extremely rocky switchover, and i’m still not entirely satisfied with it personally.

    Tens of thousands of desktops are not his (or anyone’s) personal playground, and I’m very glad that Ubuntu has learned from history, here.

    That said, if systemd ends up being all that Lennart is claiming it is, i look forward to its eventual inclusion!

  7. one year should be enough to switch to systemd. the features (especially for servers!) are very nice (eg cgroups support). So please switch now and don’t wait another year.

  8. I would also think the change a good idea; by all accounts it appears superior, Fedora has already made the switch without issue and the problem with upstart is that system administrators found more disadvantages that advantages. In addition, if Ubuntu wants to push boot speed again in 11.10 (it does seem to have regressed), this would be a great move.

  9. Please switch to it. The more guinea pigs the better.
    Is it too late to be backported into 11.04 too? It’s only been released for two days..

  10. Uh, in my blog story I made explicitly clear that this is my view on things, and I am biased. Putting “fair and balanced” in quotes in your blog story creates the impression I ever claimed it was fair and balanced. But I never did. I pointed you the fact that it isn’t.

    Also, I point out a couple of valid reasons against switching, for example the amount of work the conversion means if you come from Upstart. It’s one of the aspects one might need to take into account, but definitely not the only.

    I don’t think Mark is evil, which you appear to imply I think. It’s you who calls Mark a dictator, not me. I also have nothing against Ubuntu, I am mostly ambivalent to it. I have attended a couple of UDS. I will criticize Ubuntu when they fuck up, but I am willing cooperate whenever they are willing to do so. I think Ubuntu/Canonical needs to contribute more, but I have no doubt that Ubuntu is a cool thing as is.

    Really, whatever Ubuntu does is up to Ubuntu. I will provide those who care with the right arguments, but I don’t really care enough about Ubuntu to actually try to influence your decision making process.

    BTW, picking job visualization as an example why Upstart is good is ironic, given that systemd has been shipping that for a while, and much niftier.

    Anyway, you are barking up the wrong tree. Neither am I Ubuntu’s enemy nor does whatever Ubuntu decides really affect me.


  11. I admit I am really looking forward systemd as well. As a system administrator I’ve not really had a great experience with upstart, whereas systemd seems very promising.
    12.4 is still a fair bit away, maybe it would be enough for the inclusion of systemd? Otherwise the next server LTS that would include systemd would be 14.4 and that looks really far away…

  12. sir, say whatever you want but PLEASE do NOT quote george w.

  13. You could create an experimental package set for systemd in 11.10 and 12.04 and attract a community of people around it and the debian work to push it along side.

    By the time you’re ready to make a decision, you’ll have a much better way out if you choose to go with systemd than if you just invest nothing it in for a year.

    Help the community, help you.

  14. You are not putting upstart first “because of users”. You are putting first because of your emotional attachment to upstart.

    Having migrated a system to upstart, and another system to systemd… migration to systemd was much easier, and the users were more happy with.

    • I think not. They are putting upstart first because they simply don’t have time to get systemd working well enough for servers etc (or they don’t want to take the risk, which is fair enough) before the LTS. Therefore they are putting upstart first because *users* will most likely find problems with systemd which is not what they want in an LTS.

      Sure, migration to systemd might be easier, but since upstart is now working at least OK, if not perfectly, I for one can understand them simply not wanting to take the chance that it’ll all be dandy in less than a year’s time. Although I agree with Martin Owens that it would be good to give users the option, even if it isn’t the default ASAP.

  15. A blog post does not equal a campaign. Chill out.

  16. Someone mentioned pulseaudio! Heh Linux Sound is the one reason my recording studio isn’t on Linux!

    And seriously, I’ve had enough of this whole upstart/sysV/systemd stuff. I’m sure Ubuntu will eventually pick up on it. Get the Linux audio subsystem(s)(s)(s) sorted!


  17. Agreed. There is no hurry to adopt systemd or any other new boot-system. If things go wrong and there are critical bugs there is a risk that the whole system will suffer.
    I took a look at the comparison list and my impression is that I can achieve everything I require already, now. By applying the appropriate tools that are delivered with Ubuntu. Time will show if it is really a good idea to integrate all the functionality in one big tool.
    For most desktop users there is practically no benefit from adoption. Average desktop users do (rightly) not know about boot-systems and they do not care. And I seriously doubt that systemd will reduce my boot-time considerably. The bottleneck on most desktop systems is the slow hard-disk and not efficient boot management. Of course I would be glad if systemd would proof me wrong. 🙂
    The verdict: Stay calm, let systemd stabilize, start experimenting with it, but only adopt it when it has proved to deliver a trouble free system.

    • Inexpensive SSDs will change that (like on phones). Running simple purpose-built C programs instead of (fork, get multiple tools from hard drive to parse text, and exec) will save the need for some hard drive access, which is one of SystemD’s efforts.

      For multi-CPU machines: SystemD sets-up pipes to allow unordered startup because dependant programs don’t need everything immediately. That should reduce any sparse workload early in the boot process.

      • Upstart does this too.. fork and exec to allow unordered startup of programs. I’m not really sure what you mean that systemd “sets up pipes”. Take a look at the bootcharts for an Ubuntu system and you’ll see that there are quite a few things going on at one time.

        Getting multiple tools from the hard drive isn’t really all that big of a deal, as the only tools required to boot are tiny things like dash (100k) and mountall (100k). When you have ureadahead fetching these ahead of you, they’re going to be cached in RAM before all of your hardware is even initialized.

        The extra parsing effort for shell using dash is pretty miniscule. Meanwhile the flexibility and accessibility gained is massive. Fixing a shell script in an upstart job file is a lot easier than fixing some bug buried deep within systemd’s code.

  18. As somebody who has actually been doing a lot of the work to improve the boot experience on Ubuntu Server, I have to echo Robbie (my direct manager) and say that there is a time and a place, and 11.10 is not it.

    A year seems like long enough, but it really isn’t. There are so many things that have to be re-thought, its not even about systemd and upstart, but about how Ubuntu’s boot works on a server, and how we can make that experience as solid as users expect it to be without trashing boot speed for the desktop.

    I think systemd’s socket activation is super attractive, and will make many of the vexing issues with upstart very simple to solve. However, first and foremost, the boot must be *modeled* properly. Once this is done, any changeover will be quite smooth. The desktop boot was modeled ad nauseum in 9.10 and 10.04, and because of that, it was really solid, and really performant. But the server was left behind, with configurations like bonded ethernet, NFS, and mandatory VPN’s needing extensive fixes, and/or deep configuration by users.

    Boot and pid 1 go hand in hand, but they are not the same thing. So lets model Ubuntu’s *boot* first, then we can worry about optimizing the engine that drives it.

  19. There is no requirement to make upstart the default already. E. g. we could deliver this in several stages:

    * oneiric: prepare a package in universe which by and large works (this should not be too hard given that there is already a well-working PPA)

    * 12.04: promote it to main and make a QA effort/promise that it will work for everything in main

    * 12.10: Make it the default at least for new installs; we could keep upgrades at upstart to accomodate admins who need to port their custom upstart jobs to systemd

    * next LTS: only support upstart for installs and upgrades

  20. cgroups is new.

    systemd is new.

    Combined as a replacement for sysvinit.. it’s sort of like replacing libc with something written in perl. Not saying it isn’t possible, but there are SO many existing scenarios that systemd isn’t going to handle well. Perhaps some of the same arguments could be made about upstart…. but systemd goes a LOT deeper.

    IMHO, systemd, upstart, etc… are good examples of people trying to rewrite something that was was SO flexible, that the permutations for start up are too many to handle.. and have it be some kind of drop in replacement. I’m not saying anything that isn’t well known. You can look at the major distros, like Red Hat, SUSE, openSUSE and Fedora, etc… just see what all they are having to go through.. and notice that once again, they’re staking out their own paths on how to handle startup with the myriad of changes that systemd requires.

    Is cgroups a good idea? Yes. Is it what it needs to be? IMHO, no. There are still gaps that it needs to address.

    Is systemd a good idea? Yes (if you realize that it is more constrained than pure shell scripting) and no (if realize that your whole world is about change in a VERY radical way). Is it what it needs to be? Systemd is still BAKING. Things seemed to be getting addressed, but the designers are adamant that systemd do things “right” and “right” at the expense of drop in compatibility.

    My guess is that the first deployments of systemd will be VERY painful (and I’m not talking upstart sort of pain, likely 10x the pain… just a guess there). I predict lots of breakages… and… oh yes… need I mention… likely a lot of commercial software will not work… at least not without workarounds to their documentation (Commercial ISVs need to take heed).

    • Presumes are great about systemd. Systemd has been way wiser in its design than upstart was for migration.

      system v init scripts can still be used inside systemd without issues. Upstart lacks this “SysV services controllable like native services”. So upstart + Sysv I have manage one group of services with upstart and one group sysv style. How can you say kick ISV’s in nuts. Twice the work to get stuff done thank you upstart.

      Due to Systemd good compadiblity with Sys V. There is very little commercial software that will not work. Cases of broken are where distribution particular configuration tools are used like SUSE Yast for enabling services that at the first test past had not been updated for systemd. Yes is all the parts that debian fedora suse…. Have created custom to manage system v init is where all the hell is. For a ISV it will be great to see all that crap gone.

      Systemd has gone a shallower path than upstart. Clearly dividing the problem. Core init service starting and wrapping stuff.

      I have ready played with systemd. At worse systemd is no better than system v init. And best its well ahead.

      cgroups are not that new. Issue here cgroups can already take over most of the job of apparmor but not selinux. Main advantage cgroups are not like LSM’s like apparmor and selinux. They can always be there with or without LSM enabled. Cgroups is not fully want it needs to be yet. But systemd is a darn good start. You have to start somewhere.

      10 times the pain no. Lower performing than it should be yes due to not enough coverted.

      I really don’t get were the idea scripts are forbin comes from.

      systemd has basically done away with the rcX.d directories and the bash based processing scripts of the rcX.d.

      Replaced them with single files for each service containing more details about the service including cgroup limits.

      Also you need to look closer distributions are not staking out new paths as dividing as system v init was. Most of the hard issues is getting from the highly divided system v init solutions back to the common systemd. A few different paths will be expected.

      Hopefully as systemd matures the edge cases the core does not handle well get fixed up. Really systemd has a better chance of unify the distributions than upstart does. Reason the upstart mistake of requiring all alterations signed over.

      systemd is only scratching the surface of what cgroups can do at this stage.

      Also people like to forget at the core of system v init is a little c binary the init binary. What systemd does is expands that binary what it can do. system v init was never 100 percent script.

  21. Oh dear, yet another pathetic, whingy non-developer.

    All the other distros haven’t switched already – they’ve just stated that they plan to at some point. No one of any merit associated with Ubuntu has said Ubuntu should switch immediately. No one else outside of Ubuntu has either. Lennart has just outlined the goals and design advantages of systemd and has suggested that all Distros could benefit from these points if they make the switch EVENTUALLY.

    One of the most annoying things about Ubuntu is that it has attracted a croud of utter m0rons like yourself, who have a very basic, surface understanding of the OS plumbing but feel the need to weigh in with strong, uneducated opinions and assertions about everything.

    One of the worst things about open source is that the open discussions about the various technologies get jumped on by non-developers who think they have a clue. Stay the f*ck out of these things please. Just familiarise yourself with Gnome or KDE and stay the f*ck out of technical debates.

    Please, spend more time reading and researching and less time pretending you have a clue what is going on.

  22. Anonymous Coward

    Strictly from a technical POV, I’d rather stay with sysvinit on servers and upstart on clients. SystemD does add features, but IMO the only situation in which you do use those new features is in complex server setups, with strong dependencies among daemons located all over the server farm. That’s a bad setup in itself, regardless of which daemon startup system you use – it’s fragile.

    Upstart is too complex for my taste to be used on servers. I can’t remember who, but it was one of the Unix gurus of old, who said “when in doubt, use brute force”. That’s what sysvinit does – a brutal but extremely simple approach, suitable for servers, which are supposed to reboot seldomly and stay mostly unmodified for ages.

    Desktops or notebooks, OTOH, need to start fast, reboot often, and need to boot reliably regardless of the environment they boot into (unlike my job laptop, which waits forever and a day if there’s no wired network available at boot time). Their boot process should therefore not depend on anything external to the booting machine itself. And since desktops and laptops are supposed to start just a few services on startup (you don’t run servers on a typical workstation), upstart is just right.

    The scenarios SystemD seems to fit, as AFAICT, are continuous integration environments consisting of multiple machines which are strongly coupled together, developer workstations closely integrated into a system of development servers etc. IMO not the most frequent setup.

    Put slightly different, SystemD is a great solution to a problem which you should make sure you don’t have in the first place.

    Also, SystemD, while technically a nice implementation, strongly deviates from the Unix tradition of building systems from small components which do only one thing and do it well, connected by small shell scripts acting as glue code – which, on any Unix-like OS, should not be a big pain, since starting processes is cheap, compared to the cost of building flexibility and reliability into large systems.

  23. Anonymous Coward

    Just one more thing: cgroups are usable from both sysvinit and upstart, by using the lxc commands, so they can’t be considered an exclusive advantage of SystemD. And this way of usinroups does respect the Unix philosophy of doing one thing only and doing it well.

  24. If you really want to put users first , then you guys should work harder to set systemd as default for ubuntu, because is better

  25. Whilst I very much appreciate the move towards upstart / systemd for desktops, it’s proving a real pain for servers. The upgrade process has been nasty and given that the major reasoning behind the migration really doesn’t apply here (server firmware and hardware (scsi cards anyone) POST times are longer than Ubuntu boot times for example). Similarly they remain read at boot time and not again for often long periods of time. It’s at this point any upgrades and introduced bugs rear their heads.

    Please please concentrate on making this far more transparent, sysadmins are users too…

  26. I hope that systemd becomes the default in a future ubuntu. I think the socket based approach for depedency management is a lot more elegant than the alternatives. At the same time i’m using gnome 3 so i could just move to fedora. Pity i’m addicted to the apt-get ecosystem.

  1. Pingback: Why Ubuntu Should Continue with Upstart for 11.10 | Ubuntu-News - Your one stop for news about Ubuntu

  2. Pingback: Robbie Williamson: Why Ubuntu Should Continue with Upstart for 11.10 | Follia Digitale

  3. Pingback: Canonical nie planuje zmiany Upstart na SystemD w najbliższym czasie | thecamels.org

  4. Pingback: Ubuntu L’ocelot onirique est né?! (Ubuntu 11.10) | Club Linux Atomic

  5. Pingback: GNOME Project suffering the NIH disease | rojtberg.net

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: