Purpose. Rewrite of my EngD project documentation into a blog format. Based on the official CERN Indico presentation, the FairSHiP GitHub repository, and my TU/e EngD project repositories. This post explains the project management, architectural design, technologies, results, and the deeper “why” behind the effort. For this project, I was both the Project Manager and Product Owner (two hats is hard).

Building a Digital Spine for CERN’s Detectors

Beneath the French countryside, a machine the size of a city bends beams of protons to nearly the speed of light. This is the Large Hadron Collider, and within its experiments lie some of the most ambitious questions humanity has ever asked. What is dark matter? Why is there something rather than nothing? Could there be hidden particles, waiting at the edge of detection, carrying answers to mysteries that have haunted physics for decades?

Among the experiments poised to ask these questions is SHiP, the Search for Hidden Particles. Unlike the familiar proton-proton collisions of ATLAS or CMS, SHiP is a beam-dump experiment, designed to uncover long-lived, weakly interacting particles that slip through other detectors unseen. It hunts for heavy neutral leptons, scalar and vector portals, axion-like messengers, and even supersymmetric partners such as sgoldstinos, whispered hints of a world beyond the Standard Model.

To fuel such a search, the Super Proton Synchrotron accelerates particles to 400 GeV and smashes them into dense targets. From those impacts pour torrents of charm mesons and photons, fertile ground for hidden-sector physics. But with intensity comes enormity: event data streams at scales that quickly climb to petabytes. Without disciplined organization, this ocean of information collapses into noise.

Event data tells us what happened inside the detector. But conditions data tells us how it happened. It is the record of temperatures, magnetic fields, alignment constants, and calibration parameters. It is the quiet chronicle of every vibration and every thermal drift. Without conditions data, the events themselves cannot be reconstructed with meaning. Without it, analysis is blind.

My project was to strengthen this backbone of memory: the Conditions Database for SHiP, integrated into the FairSHiP simulation and analysis framework. The task was deceptively simple. Build an API, connect it to CERN’s software stack, and ensure that every piece of metadata could be stored, queried, and corrected reliably. But like all simple statements at CERN, the reality concealed complexity: competing technologies, international stakeholders, strict deadlines, and the ever-present demand for scientific precision.

The proof-of-concept existed already, built by an earlier TU/e team in 2017. It showed that MongoDB could serve as the storage engine, and that conditions could be structured into intervals of validity. But prototypes are not enough when millions of euros and years of data-taking are at stake. The EngD project was charged with transforming that fragile prototype into a production-level system, one that could endure not just tests, but the weight of future physics runs.

High-level Architecture

The architecture of the solution began with clarity about what a “condition” truly meant. It was not merely a number, but a value with context and time. Alignment constants shifted as detectors expanded thermally. Magnetic field amplitudes fluctuated as power systems cycled. Even air quality and temperature carried meaning for the interpretation of events. Each of these had to be recorded with an interval of validity, a bracket of time during which the value was trusted. To lose this was to lose the very ground on which the physics stood.

System Architecture

The Conditions Database therefore had to support persistence of not just raw numbers but structured metadata. MongoDB was chosen as the backbone, not by accident but by careful comparative study. Its schema-less flexibility allowed us to store evolving condition types without breaking older queries. JSON offered a human-readable interface, while indexing strategies ensured queries could scale to millions of records. SQL was retained for integration contexts, bridging traditional tools with newer paradigms.

API Flow

The API itself was written in Python, compatible with both Python 2 and 3 to accommodate CERN’s mixed environment. This decision was both a constraint and an opportunity. The constraint was that backward compatibility slowed development. The opportunity was that by supporting both, we ensured longevity and accessibility, especially as future students or postdocs would maintain the system well into the late 2020s.

Class

Comp

Around this technical spine grew a rich ecosystem of supporting tools. Docker containers encapsulated the database and API, ensuring shippable and reproducible environments. GitLab CI/CD automated integration and testing, removing human bottlenecks from deployment. PyTest enforced unit validation, while PyLint held developers to the PEP8 standard, turning code readability into a quality guarantee. AliBuild connected the project to the broader CERN ecosystem of physics simulation, compiling FairROOT and FairSHiP with reproducible precision. Doxygen generated documentation, turning code into a living reference. Enterprise Architect allowed SysML and UML models to map the flow of data and dependencies across the architecture. Each tool was chosen deliberately, not for novelty, but for fit.

Technology Stack

The process was managed as a hybrid of Scrum and PMI discipline. Sprints lasted one week, six in total, from kickoff to final delivery. User stories were broken down with acceptance criteria, tracked in a backlog, and measured by cumulative flow diagrams. The PMI side anchored these iterations with formal scope, risk, and quality plans. In practice, it meant velocity could be forecast, risks catalogued, and quality assured, while still allowing responsiveness to unexpected obstacles.

Deliverables were defined with equal precision. They included the API code, the database schema, import and export scripts, integration with the FairSHiP framework, benchmarking results, user manuals, architecture diagrams, and testing frameworks. Documentation was not optional. Knowing that many future maintainers would be transient researchers, the PMP mandated guides, plans, and checklists as first-class outputs alongside the code itself.

Performance was a central concern. The system had to deliver results comparable to or better than the baseline implementations in terms of wall time and CPU time. Benchmarking scripts measured execution time under varying loads. The metrics showed that the API met these constraints, with latency held within acceptable limits and throughput scaling effectively with database size.

Benchmark	Baseline (s)	Delivered (s)
Wall Time	1.00	1.02
CPU Time	0.95	0.96

Testing was comprehensive. Unit tests validated each API endpoint. Integration tests confirmed that conditions could be inserted, updated, and retrieved from the FairSHiP analysis chain without corruption. Stress tests simulated large volumes of drift tube parameters, confirming that MongoDB indexes could handle the load. Verification required that all code pass unit tests before merging, and validation was performed through demonstrations to CERN stakeholders weeks before project close.

Benchmark Results

Quality assurance extended beyond testing. Pair programming, code reviews, and adherence to PEP8 ensured readability and maintainability. Metrics were not just about execution time but about defect rates and code coverage. Reports documented overall team happiness, because in Agile philosophy, morale is often the earliest indicator of future defects. This soft metric was given the same respect as hard benchmarks.

Team Happiness

Risks were tracked formally. COVID-19 disruptions were noted explicitly. Domain knowledge gaps were identified, with mitigation plans involving knowledge transfer workshops. Scope creep was controlled by enforcing a design freeze after the third sprint, locking features to ensure stability and quality for the final delivery. The PMP’s philosophy was clear: if constraints are fixed in time and resources, then scope must be the variable.

Configuration management was rigorous. GitLab enforced merge requests with at least two reviewers. Versioning was tracked by the Configuration Manager role, ensuring that the master repository remained stable. The architect had final approval on merges, balancing speed with structural discipline.

Risk Management

Decision management recognized that in a high-speed project, not every decision could wait for consensus. Owners were defined for each artifact, empowered to act, but required to align their choices with the PM and architect. Escalation channels were clear, up to program management if needed. This avoided paralysis in the face of uncertainty.

Communication channels were distributed but deliberate. Mattermost was used internally, Slack linked to CERN contacts, Discord supported remote collaboration, and weekly meetings with CERN stakeholders were held via Vidyo. Email formalized external updates, and Indico archived presentations. This ecosystem mirrored the hybrid nature of the project itself: agile in the details, structured in the whole.

At the end of the six sprints, the product was delivered. The Conditions Database API was merged into CERN’s official FairSHiP repository. Documentation was handed over, including manuals, plans, and benchmarking results. A demonstration video was produced, ensuring that even those unfamiliar with the code could see its operation. The integration succeeded: FairSHiP could now import alignment data from the database and retrieve it during analysis.

Sprint	Capacity (hrs)	Velocity (SP)
I	28	26
II	20	32
III	31.5	38
IV	27.5	33
V	14	17
VI	21	25

Metrics validated the effort. Unit test coverage exceeded thresholds. Benchmarks showed no degradation compared to baseline implementations. Code quality checks passed PyLint with PEP8 compliance. Team velocity stabilized across sprints, with an average of 30–35 story points per sprint, predicting the completion of 171 points in total. Stress tests demonstrated scalability across tens of thousands of condition entries.

The cultural results were just as significant. CERN stakeholders now had confidence not only in the code but in the process. Documentation standards, QA plans, and architecture models created a reproducible baseline. The project showed that student teams could deliver production-level systems when armed with both technical and managerial discipline.

The “why” was always larger than the code. Hidden particles, supersymmetry, tau neutrinos — these are the scientific treasures SHiP seeks. But without the invisible infrastructure of condition tracking, those treasures could be buried in noise. The Conditions Database ensured that when SHiP takes data, it will do so with the memory required to make sense of it. The project was about building trust in data, trust in tools, and trust across teams.

For me as project manager, the experience was a lesson in scale and humility. I was not unlocking new physics, but I was ensuring that when the physics comes, it can be trusted. I managed risks, scope, and deadlines, but I also managed meaning — creating a structure where international teams could collaborate, where quality could be assured, and where hidden particles might one day be revealed. In that sense, the project was a reminder that even in the search for the most elusive matter in the universe, it is often the quiet architecture of project management that holds discovery together.

Final Results

Building a Digital Spine for CERN's Detectors

Project Management at the Frontier of Physics

Building a Digital Spine for CERN’s Detectors

Further Reading and References

FEATURED TAGS