|
| Sun, Sep 07th | home | browse | articles | contact | chat | submit | faq | newsletter | about | stats | scoop | 00:10 UTC |
|
login « register « recover password « |
| [Article] | add comment | [Article] |
Whenever a new kernel comes out, there's a lag time between when it's adopted by those who don't mind compiling it themselves and by those who are waiting to get it bundled in an already-tested package from the maintainers of their distributions. In part, the delay is just the result of the difference between those who live on the edge and those who stick with the tried-and-true, but could it be shortened by reducing the work that the distributions have to do to adopt the new kernel? In today's editorial, Jeff Garzik of MandrakeSoft describes the process of fitting the two together. Update: Developers from Conectiva have written to share their thoughts on the subject. Copyright notice: All reader-contributed material on freshmeat.net is the property and responsibility of its author; for reprint rights, please contact the author directly. Readying a distribution for a new release of the kernel at MandrakeSoft is, like other open projects, a constant process of refinement, testing, and interaction with non-MandrakeSoft Open Source developers. Mandrake's bleeding-edge development distribution, "cooker", is always available to outside testers on the Internet. When a new kernel appears on the horizon, the first step is to package the new development kernel in a special RPM called "hackkernel". (Mandrake uses the "hack" prefix to indicate a development/unstable version of a package.) Once hackkernel is packaged, we can begin testing the distribution with the newer kernel. Testing involves hardware testing at MandrakeSoft's labs, but, even more importantly, it involves the cooker users out on the Internet. They are our biggest resource; no amount of internal testing can replace beta testing on the Internet at large. Once the feedback starts arriving from cooker users and internal Q/A, the process of refining the distribution for the new kernel begins. This process involves the evaluation of each and every package to determine how each can best take advantage of the new kernel while still retaining full compatibility with the older kernels. The package changes which make the distribution better suited to the new kernel can come from many sources: internal developers, Open Source developers out on the Internet, and other distributions. These changes are all integrated, in patch form, into each RPM package. These updates are posted on cooker, and the process of refinement begins again. The final step in readying a distribution for a new kernel is to update the installer. Since each distribution has its own installer (for the most part), this is typically a MandrakeSoft-only affair. (However, all our updates, including installer updates, are GPLed and available freely to all.) Eventually, based on cooker user and internal Q/A feedback and schedules, the cooker distribution will be frozen into a stable form and given a release code name like "oxygen" or "odyssey." After this freeze occurs, no more new additions occur, including kernel changes. Any final incompatibilities between the new kernel and other packages are smoothed out at this point. Finally, the CDs are pressed, and ISOs are uploaded to the Internet for all to enjoy. The dialogue between MandrakeSoft and Open Source developers and users is constant. New changes to the kernel that appear on the linux-kernel mailing list and elsewhere on various project Web sites must be evaluated and tested. Bugfixes and features from MandrakeSoft must be fed back to the maintainers of the Open Source projects on the Internet. In the case of the kernel itself, this usually involves sending patches to Linus and/or the linux-kernel mailing list. Linux-Mandrake is not unlike many established Open Source projects. The process of development, integration of new kernels, and testing is a constant cycle of refinement. Editor's note:I'd like to hear any of your thoughts about the relationship between the kernel and the distributions built around it. If you're a user, does the wait for a new version of your distribution with the new kernel and features you need frustrate you, or do you just compile a new kernel on your own and hope the distribution is ready to handle it? If you're a distribution maintainer, what work do you have to do to accommodate a new version of the kernel? Are the communication channels between you and the kernel developers good enough that you get all the information you need in a timely manner? Is there anything that could be done to make the job easier for you? Do you find yourself spending much time talking to third parties who haven't made the necessary changes to their software to work with the new kernel, and convincing them to do it? Is there anything else the community could do to shorten the time between a kernel release and its inclusion in the distributions? Do you have any thoughts on distributions that use patches that haven't made it into the official kernel yet (journaling filesystem foo or USB support bar)? What about proprietary binary-only modules (video card drivers, etc.)? Thoughts from Conectiva DevelopersClaudio Matsuoka (claudio@conectiva.com) and Marcelo Tosatti (marcelo@conectiva.com) of Conectiva share their thoughts on the subject: The perfect integration of a new kernel in the distribution is, of course, one of the major concerns of every Linux distributor, and it involves expertise in kernel hackery as well as user feedback to find any problems not detected in internal testing. Internal testing just ensures that minimal quality standards are met; only the feedback of a larger circle of users can provide the real measurement of the distribution's quality. Conectiva's kernel hackery expertise is supplied by a number of kernel developers including MM guy Rik Van Riel, Marcelo Tosatti (who actually builds the kernel packages), Arnaldo Melo, Aristeu Rozanski, and others. To extend its testing circles beyond Latin America, Conectiva is currently working to mirror its daily updated snapshot distribution in as many places as possible (and get as much community feedback as possible) through the bug tracking system and mailing lists. The kernel bundled in the distribution in heavily patched. How many patches it carries is a compromise between features and maintenance costs. Keeping our tree close to the official tree avoids effort replication within the community and also it makes maintenance easier. However, there are some features which are interesting for us but not acceptable in the stable kernel tree for a variety of reasons. If these features are interesting enough for us to spend time testing and maintaining, they will be adopted. Examples are the USB backport, raw I/O, bigmem, ReiserFS, and others. This is basically the way all commercial distribution vendors work. The drawback is that once you provide USB backports, LVM, ReiserFS, or any other extra functionality, you can't remove them and say to the users that it's not supported anymore. If something happens to the main development team, it's up to the distributor to keep maintaining this functionality. As for the latest stable release, Conectiva included all these patches as well as many others such as lm_sensors, I2C, DRDB, iBCS, IPVS, VM patches, supermount, raw I/O, fair scheduling, and dxr2. Locally developed patches are always sent upstream to the official maintainers. Some of these addons require a set of userspace tools to work properly, and these tools also need maintenance. LVM, for example, has different, incompatible I/O protocol versions in different kernel releases, and the userspace tools must allow the user to boot the system using kernels with different IOP versions. In the snapshot release, Conectiva supports IOP6 (used in later 2.2 and early 2.4.0-test kernels) as well as IOP10 (used in the current 2.4). The use of wrappers and lvm-iop packages has been adopted by Conectiva and TurboLinux. Additionally, the system should still work with plain, non-patched kernels. An administrator must be able to compile her own stock kernel and still have the system working. Binary modules are not included in the kernel package. To be able to support the kernel code which is in the official distribution, we cannot accept unfixable and unmaintainable code. There may be binary modules on the additional CDs that are shipped with the distribution for user convenience, but they are not supported and the documentation is clear about that. Author's bio: Jeff Garzik can be reached at jgarzik@mandrakesoft.com. He claims to be a "not a very exciting person"[1] that MandrakeSoft pays to hack on the kernel. [1] We actually know better and have the photos from Linuxworld Expo to prove it, but will not post them if Jeff makes the suggested deposits into our bank accounts. T-Shirts and Fame! We're eager to find people interested in writing articles on software-related topics. We're flexible on length, style, and topic, so long as you know what you're talking about and back up your opinions with facts. Anyone who writes an article gets a t-shirt from ThinkGeek in addition to 15 minutes of fame. If you think you'd like to try your hand at it, let jeff.covey@freshmeat.net know what you'd like to write about. [Comments are disabled]
[»]
DIY kernel upgrades I'm used to installing a new kernel if it contains a bugfix or feature that I'm looking for. If you use LILO _and RTFM_, it's not that difficult. Having said that, I think that the packaging systems of some distributions make it very difficult to install anything that's not in the distribution's preferred packaging format. One self-installed piece of software can invalidate your whole packaging system database. I don't think I need to elaborate on that. That's why I favor distributions that do not have a packaging system that functions like a straightjacket. (such as slackware or linuxfromscratch) So my take on it is that DIY upgrades will be limited to people with at least an understanding of of what constitutes a Linux system, and which pieces depend on each other. Those that do not have that understanding have the option of learning that, or waiting for the distribution vendor to help them out.
[»]
the problem is the utilities If one is using Debian with 2.2.10 and switches to 2.2.18 there isn't much of a problem. However, switching to 2.4 is very painful, none of the modules will work (I have had to load them all manually), and there are a lot of utilities you will have to download and install by hand (iptables, reiserfs utilities, etc.) It seems to me that it would not be THAT difficult for the distributions to package a major kernel release like 2.4 with all the neccessary utilities peripheral to it. Waiting for the next stable Debian release is quite a wait, and no way I'm running unstable (besides which 2.4 isn't even in unstable). For Red Hat and friends it is really inexcuseable. I've built more kernels than you can shake a stick at, but switching major kernel revisions is a little more than I can deal with (though it depends on the machine and what it needs).
[»]
Packaged doesn't mean good I've been a Linux user since Yggdrasil... Which caused me to quickly move to Slackware :) I've been compiling kernels a long time. At home I use both Slackware and RedHat. At work, RedHat exclusivly... It's just simpler in general, but... One of my home systems uses a mulitiude of odds and ends, and the Pre-packaged RedHat kernel just hangs on my system. At work, I have a system that needs some special options in the SCSI sub-system (multi-lun support); Again, the pre-packaged kernel binaries are just not up to it and building from the kernel source RPM somehow doesn't build the module to do it. So I end up building the kernel from generic source. I don't wait for RedHat or Slackware to package up a kernel upgrade if it has something I need. I do wait for other people to run it for a while before I install the latest/greatest. The downside is some of the stuff, I'd like to play with is rather insistant about kernel version designation (redhat and 2.2.16-30) and "unsupported"... Not fun for me :( And while I'm on the topic of pre-packaged open-source software. How many of us have been bitten by badly built RPMS? My most recent foray was NIS a crucial part of the ypserver package wasn't compiled correctly, and from my experiements, CAN'T be build correctly from the source RPM! Back to the generic source! That was good for a wasted day.
[»]
Re: Packaged doesn't mean good
--
[»]
Re: Packaged doesn't mean good
[»]
I used to i used to adapt the new kernel, that day. but now i leave my box on all the time as a server and rarely reboot. so i haven't upgraded to 2.4.1 i'm still using 2.4 though. imho /most/ people aren't going to upgrade the kernel till a new release of their distro comes out. as for hoping the distro works with the new kernel, i built my own install :) distroless! -out- --
[»]
Lack of kernel updates for released Linux distributions I'd try out a new kernel version more often if there were kernel update packages for non-beta releases of my current Linux distribution. Means, I don't like to fetch packages from a beta or development version of my Linux distribution and mix these with my non-beta distribution. Actually I'd prefer ``hack''-prefixed kernel update packages which I could run with my current distribution without having to put my hands on packages from any beta distribution.
[»]
Re: Lack of kernel updates for released Linux distributions
--
[»]
Re: Lack of kernel updates for released Linux distributions
[»]
Re: Lack of kernel updates for released Linux distributions
In such situation, experimental recent kernel packages for an old distribution are as risky as using beta packages. If they get sufficient testing, they won't be recent anymore. If you don't want to build your own recent stock kernel and want new features from kernels shipped in recent distros It's usually simplier just to apt-get dist-upgrade to a newer distribution version.
[»]
Re: Lack of kernel updates for released Linux distributions
[»]
most users don't care about the latest kernel. look, i've been compiling my own kernels for about 6 years now, and i barely ever do it any more. why? because modules mean that the standard kernel (debian in my case) is capable of being dynamically configured to do most anything i'm gonna want. there are pretty much only two cases where i recompile my kernel these days: - because i prefer using laptops, sometimes i need a patch or a bleeding edge kernel for driver support. - on my server for security reasons. i like to compile in openwall, frees/wan, sub-domain etc for security reasons and these aren't included in any bundled kernel that i'm aware of. these reasons are specific to me and my needs. most users have hardware that is much better supported then my laptops, and few will run servers let alone servers with fairly strenuous security measures. if something can be done to make the process of implementing a new kernel easier for distributions then great. however i hardly see this as a pressing issue. after all isn't the sort of integration the reason that distributions appeared?
[»]
Re: most users don't care about the latest kernel. Well, I hope all Linux users won't think the same that they don't need to have user friendly interfaces and programs ! Otherwise it will remain a software for programmers, people who like to speak with machines. Saying that people need to learn how their computer work and how to talk to it is like if scientists wouldn't give out their knowledges... Or a company building a new technology but without saying how it's like. Information retention ! Otherwise, I tested kernel 2.4.0-ac10 which was instable on my computer (freezing sometimes) so I had to go back to the previous one. I'll wait for it to be stable (and integrated in the next release of Mandrake (or any other distro)).
[»]
Linux Kernel Database Proposed to Make this Easier One thing that can speed both the development of new kernels, and their adoption when they are released is more widespread and more effective quality assurance of the kernels during development and immediately after released. The distros often maintain full-time staff to do QA, their resources are necessarily limited. There is no way any company can maintain staff and enough different machines to reproduce all the configurations a new kernel will be used on in production once released. To encourage more regular users to participate in Linux kernel testing, and to make the process of getting their feedback easier on them, I have proposed the Linux Quality Database that will provide a web form backed by a powerful database for capturing configuration information and making bugs easily searchable by kernel developers. It is a big project, and I need help before I can start, particularly from a database architect who can advise me on the schema for the database. In the meantime, there are articles being placed on the site to guide you in the process of testing the kernel and writing better free software in general. --
[»]
Where are the warriors today? If you're a user, does the wait for a new version of your distribution with the new kernel and features you need frustrate you, or do you just compile a new kernel on your own and hope the distribution is ready to handle it? Compile and let it fly, after screwing up your distribution with an incompatible kernel you can always run windows when you want to be productive again. :) In all seriousness, the GNU/Linux OS is NOT ready for people who would get frustrated and unwilling to download a newer stable kernel with the features they need. So if you are waiting around for your distro to include the kernel, just run windows to get the features you need.
[»]
Re: Where are the warriors today?
--
[»]
Re: Where are the warriors today?
--
[»]
Re: Where are the warriors today?
--
[»]
Re: Where are the warriors today?
[»]
Re: Where are the warriors today?
[»]
Small trees! Bonsaaaai! ...or something like that...
[»]
Re: Where are the warriors today?
[»]
Re: Where are the warriors today? off the bat, let me say that I am learning Linux and computing. A friend helped me put in RH 5.x some time ago and I kept it as a second OS for a long time. I remember my first year of Linux. I spent most of my modest computer time configuring and checking things out. I discovered MDK and wow! I came on board. Now I run Linux as my OS-of-choice at home. I am still learning; but often I put off the tinker gloves and decide to USE my computer to surf, write, etc. Nothing is a culture of one (except maybe insanity). For the expert, and the character with way too much time on his/her hands, for those virtual individuals, I am sure spending the next 36 hrs finding out why some item did not fly is the first choice in a worthwhile life--and that's great. But the world is always more diverse than some people are willing to admit. (The existence of Mandrake itself speaks against a 1 class society of warriors.) I need you guys who are the warriors of Linux, but please don't dispise the rest of the tribe: after all you are defending the village from that evil invader who, at this moment, is about to lay seige to to this village/(ok town): get ready folks! the rape and pillaging of Linux has yet to begin... what! you think evilB will just donate his money to charity and lose disappear in an infinite smile.
|