Architecture

This document details different components of the code base. The intention is to provide a simple to explain but a high-level description of the projects goals and plans to each of these internal mechanisms.

flowchart TB
    pkg[rypper] --> op1{install?}
    pkg[rypper] --> op2{distro-upgrade?}
    op1 -->|Yes| MIRROR[(Mirror)]
    op2 -->|Yes| MIRROR
    op1 -->|No| E[END]
    op2 -->|No| E[END]
    MIRROR --> resp{RESPONSE}
    resp -->|OK| pkg
    resp -->|ERR| E
    subgraph "rypper libraries"
    a1[rypper util] --> s1[rypper core]
    b1[rypper tui] --> s1
    c1[rypper cli] --> s1
    d1[rypper reader] --> s1
    s1 --> pkg
    end

CLI (rypper-cli)

The cli is the frontend for running commands behind the scenes. The goal is to make it readable, and to make it as close as zypper's cli commands.

The following commands that are planned to be close to zypper are the following:

  • install (aliased as in)
  • remove (aliased as rm)
  • update (aliased as up)
  • dist-upgrade (aliased as dup)
  • add-repo (aliased as ar)
  • remove-repo (aliased as rr)
  • add-lock (aliased as al)
  • remove-lock (aliased as rl)
  • modify-repo (aliased as mr)

See the Cli and Command enums for more planned commands.

Other commands related to services seems to be not planned for now.

Reader (rypper-reader)

The reader handles all the metadata that is fetched from upstream. These includes but not limited to the following:

  • Repo files. They are actually in ini format. Example:

    [Publishing]
name=Publishing Tools and Libraries (openSUSE_Tumbleweed)
type=rpm-md
baseurl=https://download.opensuse.org/repositories/Publishing/openSUSE_Tumbleweed/
gpgcheck=1
gpgkey=https://download.opensuse.org/repositories/Publishing/openSUSE_Tumbleweed/repodata/repomd.xml.key
enabled=1

  • XML Metadata from repo files. They contain data of all available packages in that repo.

  • Signature verification. They use GPG. I cannot avoid it though ☺️

Utilities (rypper-utils)

The utilities contains all helper/handler functions and stuff that are used around the project that may not fit in some rypper libraries.

Core (rypper-core)

Core functionaliy that bundles all other rypper libraries. The logic are finally wrapped around in this library.

NOTE: It's a good idea now to read the below section about mirrors, else this section may be confusing.

There are three major tasks that a library like this must achieve.

  • Download and retrieve metadata about packages
  • Compare the repo metadata and system metadata to determine what changes are required to satisfy the request
  • Perform each action in turn

Each of these have some overlap in functionality but we'll go through them all at a high level.

Download repodata

  • Determine the list of repositories from /etc/zypp/repos.d
  • For each repo where enabled and refresh are true
    • Download repomd.xml from the url
    • validate the signature of this file (repomd.xml.asc)
    • parse repomd.xml to discover the current metadata databases (sha256sum-metadat.xml for example)
    • download these files
    • validate the signature of the files
  • Persist the repo data in /var/lib/zypp somewhere for later local use.

Compare the repo metadata

Given some requested action (for example install x)

  • Determine if x exists on the system from system metadata (rpmdb or similar)
    • If the action is already satisfied, cease
  • Find which repo provides x (this is where repo prio is applied)
  • Create a transition to satisfy the action
    • Determine from that transition if dependent transitions are required
    • Examine dependent transitions for their requirements
    • repeat until stable graph of transitions is achieved.
  • These transitions are now an ordered list of actions to perform to achieve the desired state.
  • Examine the transitions for boolean satisfiability

EXAMPLE

rypper install x

repodata.find_provider(x);

if rpmdb contains x {
    // already installed
    if x.vendor != installed.vendor ||
       x.version > installed.version ||
       reinstall {
       // Then we need to install this.
    }
}

transitions = []
unresolved_transitions = []

transitions.push(T::Install(x));
unresolved_transitions.push(T::Install(x));

while let Some(t) = unresolved_transitions {
    for dep in t.has_deps() {
        // find the provider of the dep if an install for example.
        transitions.push(t);
        unresolved_transitions.push(t);
    }
}

// transitions is now a reverse ordered list of actions, ie FILO/LIFO queue.

system_rpms = ...;

// check that the system rpms are boolean satisfied.

for t in transition.reverse() {
    // can t be added to system rpms without breaking boolean satisfiability?
    // if not, err.
}

// can proceed!

Perform the actions required.

There are realistically only two actions - install and remove.

  • for each transition
    • if the transition is an install AND the file is not already cached
      • queue the file for download
  • for each file in dl queue
    • request metalink from mirrorcache
      • from the set of mirrors, sort them randomly
        • download from the mirror
        • if error, go to the next mirror.
  • check each downloaded package signature matches.
  • decompress each package's contents (either into ram or to disk)
  • install each package in order of the transitions (remember, LIFO).

Performance

Initially you want to make all of the above sequential - one step at a time. Then we want to examine how to achieve concurrency and parallelism.

Remember:

  • concurrency - doing N things at a time on one thread
  • parallelism - having N threads available to do tasks

From the above listing, there are some obvious places where this can be employed.

  • Repo downloads - each repo dl can be it's own task

  • Compare - checking if X is present in rpmdb and which repo provides X

NOTE: the dep solver probably can't be made concurrent.

  • Perform - process each item in the dl queue as a task
  • Perform - check multiple signatures concurrently
  • Decompress multiple packages concurrently

Once you add in concurerncy you also have some extra challenges, mainly around communication to the user. The way to manage this is you have a queue of events that are sent to an interaction thread for display.

┌─────────────┐                      ┌─────────────┐
│             │                    ┌─┴───────────┐ │
│             │                    │             │ │
│             │                    │             │ │
│             │                    │             │ │
│             │                    │             │ │
│ Interaction │       Queue        │             │ │
│Thread / Task│◀──────event────────┤ DL Workers  │ │
│             │                    │             │ │
│             │                    │             │ │
│             │                    │             │ │
│             │                    │             │ │
│             │                    │             │ │
└─────────────┘                    │             ├─┘
                                   └─────────────┘

In psuedo code:


let broadcast_tx, broacast_rx = ...;
let event_q_tx, event_q_rx = ...;

let x = num workers;

let sem = sem::new(x);
let workers = Vec ...;

for i in 0..x {
    let sem_guard = sem.acquire();
    workers.push(task::spawn(async move ||{
        // 
        let _ = sem_guard;

        while let Some(task) = dl_queue {
            // Get the metalanik
            // shuffle
            // download.

            // during the download you can send events like
            event_q_tx(E::DLStart { name, size });

            ...
            event_q_tx(E::DLProg { name, have, size, rate });

            ...

            event_q_tx(E::DLEnd { name, size, rate });
        }
        
        // Sem guard drops here. When all sem guards drop, this releases the select
        // select loop since it wants to acquire even sem guard that the workers had.
    }));
}

select! {
    _ = signal.ctrl_c => {
        broadcast_tx(stop);
    }
    event = event_q_rx => {
        // Do something with this event
        // Allows back signalling of errors.
    }
    _ = sem.acquire_many(x) => {
        // This is waiting on *all* dl tasks to drop their semaphore.
    }
    
    
}

broadcast_tx(stop);

for handle in workers {
    handle.join().await
}

Now the fun part here is how you want to display the information from the events! You'll be downloading many things concurrently so the traditional zypper "one at a time" won't work. Some ideas are:

Just show the overall status rather than each package individually.

[spinner] rate <sum of last rates>/s    have / need

Another idea is to "fixate" on a package at a time.

If you get a DLStart { name } you start showing the "download" line like zypper does. For each other DLStart you get, you queue them locally. Then each event you get, if it's for name you show it, else you queue it in the bg. Once you get the DLEnd for name, you can then dequeu from your local queue and fixate on the next package from it's DLStart until you get to the end etc. (You may need a few little queues for this but it'll work). In the local queue you only hold on to DLStart and one related event from name (either prog or end).

Effectively this will "bunch up" events so a large blocking DL will show it's progress, while smaller ones completing in the BG will appear to finish rapidly once complete. It then means that if another big package starts in the background, once the first one finishes as you start to display the second you'll only see it's current progress so you'll just from large_a 100% to large_b 45% or similar.

Mirrors vs Mirrorcache

Packages are born from OBS. No one can explain how it do, but it be. Once created, the package is placed on a hidden primary mirror source which has the extremely hidden url https://downloadcontent.opensuse.org (dlc.o.o is a common abbreviation) . In addition there is a supporting cache-proxy of this server called https://downloadcontent2.opensuse.org (dlc2.o.o).

There are now two important tasks that occur. First, the packages are distributed from dlc[2].o.o to mirrors via an assortment of synchronisation mechanisms. Second, mirrorcache (terribly named...) must index the content of those mirrors to determine what is available.

When we say "mirror" we mean some form of volunteer infrastructure that uses it's storage and resources so store and supply the packages to clients. For example in Australia there is mirror.aarnet.edu.au which acts as a mirror of the dlc.o.o content. There are many mirrors around the world, and these are what the end user system will contact to fetch its packages from. You can consider this a "but we have CDN at home" CDN.

The Migration of the Packidge

Packages are distributed from dlc[2].o.o in three primary ways. The use of rsync, rmt or via a chained mirror (such as https://github.com/Firstyear/opensuse-proxy-cache/).

These effectively take two different strategies. rsync and rmt both proactively supply all packages to the mirror. This is how many traditional mirrors work, such as the aforementioned aarnet mirror. proxy-cache uses a reactive strategy where on first request to the mirror the package is downloaded from dlc2.o.o and then cached. There are pros and cons to both approaches.

Proactive

In the proactive case, the mirror may lag behind dlc[2].o.o. This can manifest to users as missing metadata or missing packages. Additionally, some mirrors may not have atomic synchronisation of their content meaning that a package can be in the process of being rsynced as the user attempts to download it, leading to an error. Another form of this is if the mirrors rpm metadata is updated before the packages are, leading to a missing file error.

Finally, not all packages are actually used or installed, meaning that the mirror may be hosting files for content that will never be downloaded (which is a penalty to the bandwith between dlc[2].o.o and the mirror, but also the storage of the mirror). A prime example of this is OBS home repos that may only ever be used by a single person and are not relevant to the broader install base.

The benefit of this approach is that you only have to transfer the package to the mirror once. It also means that the mirror has a full copy of all content even if dlc2.o.o is not available.

Reactive

In the reactive case the mirror never lags behind dlc[2].o.o because it will always query every file name against dlc2.o.o meaning that all updates immediately are available. This also prevents the atomic synchronisation issue. Since the mirror only caches what is requested, then the storage will only be used for exactly packages that are consumed by users. This allows the mirror to cache even home repos and obscure repos that would otherwise not be serviced by a local mirror as there exists user demand for that content.

The drawback is that on first request of a new file it must be transfered from dlc[2].o.o which causes latency and a lower download speed. Only on the second or subsequent requests is it served at a local speed. This means if a file is requested once and only once then the reactive strategy is no better than going to dlc[2].o.o instead. Where this works well is when the package is requested a lot. As an example, mirror.firstyear.id.au (m.fy.id.au) has an 85% cache hit rate meaning that on average for each package requested, 85% of the time it can be found and served from the local cache without needing to go to dlc[2].o.o.

The other major drawback is that this system requires dlc[2].o.o to be online in the case of a cache miss. If dlc[2].o.o is offline, then the mirror is unable to serve new content (and can only serve content that is already cached.

The Discoberdy of the Packidge

Now that we have mirrors that have all our freshly migrated packidges, we need a way to inform users of where to find these. Mirrors can be added, removed, renamed etc. This makes distribution of a static mirror list to clients "challenging" especially because the content of those mirrors needs to be understood if it's up to date and complete.

For this purpose a system called "mirrorcache" exists. This tool is configured with nearby mirrors and is able to index them to understand what content is available.

The "primary" instance of mirrorcache is https://download.opensuse.org/ and the file listing that you see is generated by indexing of mirrors that are configured in this mirrorcache instance ( https://download.opensuse.org/report/mirrors ).

When you select to download a file, one of those mirrors is chosen based on "factors" and you are redirected to download from that mirror directly. This is why mirrorcache is a redirector - it points the user where to go to find what they want.

Mirrorcache also has a special behaviour - if you submit your request with an accept-contentype header of "metalink" instead of sending you a 301 redirect, you are sent an XML file with the list of all mirrors that contain the file you requested. This allows the client to make it's own decision about where to download from.

Since clients need to access this redirector frequently during an operation, it's important that these are in close proximity. There are multiple mirrorcache instances around the world in:

  • CZ - https://download.opensuse.org/
  • AU - https://mirrorcache-au.opensuse.org/
  • US-E - https://mirrorcache-us-east.opensuse.org/
  • US-W - https://mirrorcache-us-west.opensuse.org/
  • JP - https://mirrorcache-jp.opensuse.org/
  • BR - ???

Putting it all Together

  • OBS births a packidge
  • The packidge walks to dlc.o.o
  • The packidge is sent via rsync to a mirror
  • Mirrorcache indexes all the mirrors
  • zypper requests repodata from mirrorcache
    • mirrorcache provides an xml list of mirrors that have the repodata
  • zypper parses the repodata to discover what packages exist
  • zypper request a package from mirrorcache
    • mirrorcache provides an xml list of mirrors that have the package
  • zypper downloads the package and eats it.

What about the CDN?

The mirrorcache system described above is a CDN - it distributes content!

Recently an experiment with a commercial CDN started (cdn.opensuse.org I think). Rather than use the mirror system above, the CDN acts as a like the reactive proxy with instances all around the world. So when you contact cdn.o.o it has it's own internal caches and processes to retrieve content back to dlc.o.o. This means that the client like a user or zypper only need to download directly from the cdn and "it sorts out" all the caching and locality.

Mirrors are essential for every Linux distribution. In openSUSE, packages, software and libraries are built around from what is called as Open Build Service or for short, OBS (not the Open Broadcasting Software). Packages go there to be built and thoroughly checked through openQA. If a package successfully builds in OBS, it will then be pushed into a Mirror such as https://download.opensuse.org/. Once there, it can be downloaded as an RPM package using a package manager e.g. zypper.

Therefore, a mirror is basically a "place" where packages are hosted for a Linux distribution to be installed from or updated. But what is openSUSE's MirrorCache interface?

As defined by their description of their project on GitHub, it's a redirector. There are many mirrors that host packages for various distributions around the world including openSUSE's. An example to check if you are in a mirror redirector is to run:

This will give you a metalink version 4 file. It lists the mirrors that are available and closest to you based on region

curl -H "Accept: application/metalink4+xml"  https://mirrorcache-au.opensuse.org/tumbleweed/repo/oss/repodata/repomd.xml

curl -L -H "Accept: */*"  https://mirrorcache-au.opensuse.org/tumbleweed/repo/oss/repodata/repomd.xml

curl -H "Accept: */*"  https://mirrorcache-au.opensuse.org/tumbleweed/repo/oss/repodata/repomd.xml

CDN (RIS)

New architecture from openSUSE? or for openSUSE?