Interview: White rabbit

| October 20, 2022

Interview: White rabbit

by the Open make team, Javier Serrano and Amanda Diez Frenandez. Copyright to the authors, distributed under a CC-BY 4.0 licence.

Sections:

Banner image: white rabbit logo, By CERN, distributed under a CC-BY-SA 4.0

Interviewee: Javier Serrano (CERN) & Amanda Diez Fernandez (CERN)

Interviewers: Robert Mies (TU Berlin) & Moritz Maxeiner (FU Berlin)

Transcription and editing: Diana Paola Americano Guerrero, Robert Mies, Moritz Maxeiner & Julien Colomb

screenshot of the interview

Screenshot of the interview.

At CERN, the European Laboratory for Particle Physics, researchers perform experiments in which particle beams are passed through a series of increasingly higher energy accelerators, until they reach the LHC, the Large Hadron Collider, which is currently the biggest accelerator at CERN. There, the beams are made to collide so the physicists can study the outcome.

From an engineering perspective, this requires a distributed real-time system to ensure all components are synchronized at sub-nanosecond accuracy - something previously existing solutions were unable to provide. The project to develop the necessary hard- and software as open source, spearheaded at CERN by a team of electronics designers and Linux kernel developers around Javier Serrano (section leader), became known as “White Rabbit”. Amanda Diez has joined the project recently to increase the communication efforts and further drive the uptake of the technology outside CERN.

The White Rabbit in a nutshell

White Rabbit equipment

Photos of the White Rabbit team and equipment, OPEN-PHO-TECH-2020-002-14, source and copyright CERN

  • Project start: 2006
  • Core development team size: 5-10

Hardware products

The White Rabbit is a distributed real time system to ensure all components are synchronized at sub-nanosecond accuracy.

As soon as you want to go into the microsecond realm, then you need hardware support. So, we use optical fibres with two different wavelengths of light going in different directions.

The main product is the White Rabbit switch. This is something that you need in every White Rabbit network. Then, for people who want to design White Rabbit nodes, we tell them a recipe. No two nodes are equal. People have different needs.

White Rabbit is an extension of Ethernet. You can do anything you can do with Ethernet and more.

Hardware maturity

These things are in the market, but they could be improved for sure.

Rebuilds

There are, I think, four companies that commercialize White Rabbit switches. So at CERN, we don’t make them, we just buy them from these companies.

If the purchasing people tell us that we should be looking for providers in this or that country, it is a much simpler prospect with White Rabbit than with any proprietary hardware.

The project

Project start

In 2005-2006, we started discussing about the next generation of timing system at CERN.

We wanted to get an atmosphere, an environment for hardware developers, that would be similar to the software developers who were working with the Linux Kernel community.

We decided that we would use standards whenever possible, and when not possible, we would extend the standard. We also decided that we would open source the whole thing and involve commercial companies from day one. These were all very basic decisions in the beginning of the project that ultimately played a key role in the success of White Rabbit.

How did it all start with the White Rabbit project?

Show answer

Hardware importance

As soon as you want to go into the microsecond realm, and we go into the nanoseconds, that is six orders of magnitude better than Network Time Protocol, then you need hardware support. So, we use optical fibres with two different wavelengths of light going in different directions.

White Rabbit is an extension of Ethernet. You can do anything you can do with Ethernet and more.

Today, what is the core benefit of this project? And how does the hardware fit in the overall project?

Show answer

Funding

A lot of the funding comes from CERN. There was a grant from the Spanish government. Two Spanish companies worked on the initial design of White Rabbit, one on the hardware side and one on the software side. And that was a very good initial push. Then other people joined.

White Rabbit has been going on for something like 13 years. The involvement at CERN has ranged from three or four FTEs at the height of the development to one, one and a half FTEs at the lowest parts of development.

How is it funded, overall?

Show answer

Can you give an overall estimate over the years, how much funding you have received?

Show answer

Project process

It was a very clear program of work, and we handled it in an open source way inspired by open source software. It was quite informal in the end.

We had annual meetings where everybody would show up and show their latest work, and synchronize with the rest face to face.

Could you describe the overall process in the project?

Show answer

Major issues

Some of the issues were brought about by people who utilise White Rabbit in an environment which was not what we anticipated at CERN. That was actually a very good thing, because we had bugs that we had not noticed at CERN. They made those improvements, fed it to us, and they merged it with the standard open source projects. So, that was very nice.

That was one of the organizational challenges: how to coordinate all the actors and how to motivate them to play the game, to stay within the main project and also to open source their improvements.

What major issues have you come across in this project and how did you resolve them? It’s a long stretch of time, I guess.

Show answer

Decision making

The project works insofar as people find it useful to play the game and to contribute to the main branch that we are maintaining.

We’re starting to put in place with Amanda that we should communicate more broadly the advantages of doing so.

People just [contribute to the project] if and when they consider that this is useful for them.

There are colleagues who are going to test and participate in the validation. But most of the time, the decision happens just at an individual level.

How were the decisions made within the project?

Show answer

Who essentially decides what’s good enough to be merged back into the main branch? Is there one person whose job that is or a committee who decides this?

Show answer

The Hardware

Hardware components

The main product is the White Rabbit switch. This is something that you need in every White Rabbit network. We’re in version three now, and we are preparing version four.

And then, for people who want to design White Rabbit nodes, we tell them a recipe. There’s a reference design, because no two nodes are equal; people have different needs. For example, somebody working on cosmic ray detection or in the stocks market might be interested in a time-stamping node.

There are, I think, four companies that commercialize White Rabbit switches. At CERN, we don’t make them. We just buy them from these companies.

What are the hardware products that you have developed as part of this project?

Show answer

If you were to rate the maturity of the hardware components that you’ve developed, like prototype, demonstrator or market-ready product, how would you rate them?

Show answer

You said the GSI Center has been using one of those switches and people in China as well. Have they reproduced those switches themselves? Or did you build them at CERN and send them?

Show answer

Research outputs

Academic outputs

Hardware, software, and documentation, and papers. The hardware itself is available from commercial companies, which we always thought would make things scalable.

There has been a lot of publications. Not only about the technical features of White Rabbit, but about the White Rabbit phenomenon in itself, like, people studying the open source hardware phenomenon and the White Rabbit community

What were the envisaged outputs of the hardware development in terms of documentation, prototypes, journal publications, and these kinds of things?

Show answer

Publication strategy

We present and publish papers in conferences on big physics facilities and synchronisation.

In its latest incarnation, the Open Hardware Repository [the platform where the hardware documentation is] is nothing else than an instance of GitLab Community Edition. It would be nice to have things more indexable, easier to find. So, I see space for improvements in the way these projects are documented and presented to the world. It is true that open sourcing something needs an extra effort on quality.

Where did you publish these things? I mean in what journals and did you put the hardware files on your own systems or on other systems?

Show answer

How did you publish the documentation, including the CAD files and all the other digital parts that describe the hardware?

Show answer

Was there anything that you didn’t publish? And if you didn’t publish it, why not?

Show answer

Successes and Failures

It had an impact way beyond what we expected, beyond the physics realm, like telecom, energy distribution, or stock market. It was also very interesting because it brought other types of feedback to the project that improved it even more.

To align our visions with the competitive interests of partner companies was not so successful. This kind of friction, we did not anticipate. We’re trying to see how we can fix it in the new White Rabbit Collaboration that we’re preparing

What would you say was successful about this project and what wasn’t?

Show answer

Participants

Personal gain

At CERN, we have a mission to conduct research in fundamental physics, but also to share the things we do with the rest of the world. So, the mandate of CERN suits me perfectly. And in 2008, at the start of this project, I had been at CERN for 10 years already. I had reached a kind of maturity at an engineering level that allowed me to innovate in my domain.

What brought you to work for this project at CERN ?

Show answer

Core team and community

The key contributors to White Rabbit would be a few tens of people. It would not reach a hundred. In terms of users, it’s many more than that.

I think a big part of that success, in hiring and attracting talent, has been the open source nature of White Rabbit.

The obvious advantage for everyone is they got something that none of us could have developed alone.

How many members worked on the project over the course of time?

Show answer

How did you find suitable project members? With such competences.

Show answer

Work Coordination

Most, if not all of the teachings from the open source software world transpose quite nicely: respecting people, treating them as first-class citizens, making them feel part of a community and valuing their work. All these things that people do well in successful open source projects.

We created Frequently Asked Questions for every project, so that we don’t end up answering the same questions over and over in the forums.

How did you coordinate the work of the key contributors?

Show answer

Can you say how the members have benefited from their work in the project?

Show answer