Index of /sites/linuxsoft.cern.ch/slc/archive/datascientific/scientific/7x/contexts

Icon  Name                                                               Last modified      Size  Description
[PARENTDIR] Parent Directory                                                                        -   
[DIR] example/                                                           2018-07-03 21:28    -   
[DIR] fermilab/                                                          2016-06-08 17:46    -
Understanding the Scientific Linux Context Framework

New with Scientific Linux 7: Scientific Linux Context Framework!

1. Introduction

Starting with Scientific Linux 7 it is even easier to deploy use case specific customizations.

The context framework is designed to scale from smaller projects to collaborations. Through leveraging package repos and groups, specific tools and configurations can be provided in parallel with any on-site systems management procedures or the packages can be used as a payload by those tools.

Physical locations, scientific applications, or experiments can provide the tools they use in a traditional package repository. Interested users can then quickly locate specific behaviors and add them as needed.

Tip For users familiar with previous spins, such as Scientific Linux Fermi (SLF), these are now packaged as a Scientific Linux Context.

2. Installing a Context

There are several ways to install a Scientific Linux Context:

  • Kickstart (at Install)

  • Boot Line at Install Time

  • Interactive at Install Time

  • Interactive After Installation

2.1. Kickstart

Using an Anaconda addon, the settings can be specified directly in the kickstart file.

%addon org_scientificlinux_contexts
example
%end

Additional details can be specified:

  • The --testing argument enables the context testing repo for all enabled contexts.

  • If url is omitted the default is assumed.

  • If testing is omitted, the default (off) is assumed and the context’s testing repo will not be enabled.

  • The entry is a pipe delimited list, url|name|testing.

Note The --testing argument will force the testing repo to be enabled for all contexts.
This means our first and second examples are functionally equivalent.		 
%addon org_scientificlinux_contexts --testing
example
%end
%addon org_scientificlinux_contexts
example|testing
%end
%addon org_scientificlinux_contexts
http://url|example
%end
%addon org_scientificlinux_contexts
http://url|example|testing
%end
%addon org_scientificlinux_contexts
http://url|example|testing
secondexample
%end

2.2. Boot Line at Install Time

Using an Anaconda addon, the settings can be specified directly on the boot line.

  • context.testing is identical to --testing for kickstart

  • context.enable= is a comma-separated list of entries identical to kickstart 

    vmlinuz context.enable=example context.testing
    vmlinuz context.enable=example|testing
vmlinuz context.enable=http://url|example
vmlinuz context.enable=http://url|example|testing
vmlinuz context.enable=http://url|example|testing,secondexample
Note The samples here should be equivalent to the Kickstart examples above.

2.3. Interactive at Install Time

The Scientific Linux installation media includes an installation screen for CONTEXT where individual contexts can be located and selected.

The CONTEXT menu option is available in both GUI and Text mode. Though, Text mode is significantly more limited than the GUI.

2.4. Reloading The Context

You can reload the context packages by running:

/usr/libexec/yum-conf-context-example/reload.sh

Yum’s default configuration is to load mandatory and default packages. However, this script will honor whatever you’ve set for group_package_types within /etc/yum.conf.

2.5. Reading the Local Yum Database

You can review state via yum group info example.

$ yum group info fermilab
Group: Packages to support the Fermilab Context
 Group-Id: fermilab
 Description: These are packages for the Fermilab context.
 Mandatory Packages:
   =SL_gdm_no_user_list
   =fermilab-conf_system-logger
   =fermilab-util_ocsinventory
 Default Packages:
   =fermilab-conf_email-gateway
   =fermilab-conf_kerberos
   =fermilab-conf_ssh-client
   =fermilab-conf_ssh-server
   =fermilab-conf_timesync
 Optional Packages:
   fermilab-util_makehostkeys

2.6. Interactive After Installation

Installing a context after system installation requires a few steps:

  1. yum install yum-conf-context-example

  2. sleep 3s

  3. yum update

Note The sleep statement is required as yum-conf-context-example is performing some background tasks which must complete before the yum update.

3. Making Your Own Context

Each Context is about enabling a specific computing goal. Whether the goal is computing on-site at Fermilab or performing CMS analysis does not really matter so long as it is specific.

An example Context named example is provided to help demonstrate the framework.

Note Nearly all content within the Scientific Linux Context Framework is provided via RPM packages.
Anyone interested in making their own Context should be familiar with creating RPMs.

3.1. Naming Your Context

Context names have a few rules:

  • The name may only contain the following:

    • lower case Latin characters

    • integers

  • For clarity, the name should be obviously associated with the computing context

Examples:

  • cdfrun2

  • cms2015

  • fermilab

  • sllive

3.1.1. Defining Your Scope

The behaviors your Context provides should stop at the earliest clear endpoint.

This can be somewhat nebulous, but, if built correctly, your Context should be able to co-exist with another Context.

Note It would not be out of place for a CMS Context to depend on a CVMFS Context, rather than maintain that software stack internally.

When building your behaviors, you can always add new functionality. It is often difficult to remove features and transition to alternate software sources.

Tip Ask yourself, “Based on the name of my Context, does it make sense to add this item?”
CPU Architectures

Contexts are not required to support all architectures provided by Scientific Linux.

Based on the scope of your context can limit what architectures your packages support.

3.2. Types of Behavior

Each customization you add should serve a narrow purpose.

We believe the following behavior types should cover all expected use cases:

  • base

    • A base package should depend on other packages providing the basic feature set for a given, often complex, task. It should contain no files, have no scripts, and provide no triggers.

  • conf

    • A conf package should provide the configuration for a defined behavior.

      • Whenever possible a conf package should utilize any service.d directories. In that case, the configuration file should be correctly tagged as %config or %config(noreplace).

      • If an existing configuration file, provided by another package, must be modified, this should be performed via a RPM trigger.

  • logo

    • A logo package should contain any relevant artwork.

  • repo

    • A repo package contains a yum compatible RPM package repository. The recommended default state of this repository is enabled, but this is not a requirement.

  • util

    • A util package contains an application, a set of utilities, or a program. It may also contain its configuration files.

3.2.1. Scripts and Triggers

Any script or trigger must only utilize commands available within TUV’s base package set.

If at all possible, the commands and behaviors should be those from TUV’s initial release (#.0). Tools and behaviors from later updates may only be used if the RPM package Requires are correctly filled out to note the restriction.

Any triggers altering configuration files (provided by conf packages) may only execute under the following conditions:

  1. The initial installation of either the trigger package or the conf package

  2. The conf package is being upgraded

Note A trigger provided by a conf package should not alter files under any other conditions.

4. Context Directory Structure

contexts
|-- contexts/.contexts.json
`-- contexts/example
    |-- contexts/example/README.html -> docs/index.html
    |-- contexts/example/debuginfo
    |   `-- contexts/example/debuginfo/repodata
    |-- contexts/example/docs
    |-- contexts/example/mirrors
    |   |-- contexts/example/mirrors/base
    |   |-- contexts/example/mirrors/debuginfo
    |   |-- contexts/example/mirrors/SRPMS
    |   `-- contexts/example/mirrors/testing
    |-- contexts/example/RPM-GPG-KEY-context-example
    |-- contexts/example/SRPMS
    |   `-- contexts/example/SRPMS/repodata
    |-- contexts/example/testing
    |   `-- contexts/example/testing/x86_64
    `-- contexts/example/x86_64
        `-- contexts/example/x86_64/repodata
\.contexts.json

Is the toplevel metadata file which contains a single JSON array naming the contexts.
There is only one .contexts.json file at the top of the contexts tree. It is not tied to any particular context.

README.html

This file is optional. If you create a README.html, it must be a link to some file within the docs/ directory. Exactly which file it points at is up to you.
The README.html should serve as a quick start guide for your users.

RPM-GPG-KEY-context-example

Contains the GPG key, or keys, used for signing all RPMs within the context.

docs

Contains documentation on this context - possible usage suggestions, kickstarts, or other relevant documentation.

debuginfo

Contains all the debuginfo for all binary packages within the context repo. This includes debuginfo for any testing packages.

SRPMS

Contains all the source rpms for all packages within the context repo. This includes SRPMS for any testing packages.

testing/$basearch

Contains all the testing rpms, for a given architecture. Even if your context does not utilize testing at this time, this repo is required. There is no problem with it remaining empty.

$basearch

Contains all the rpms, for a given architecture.

mirrors/base

Contains a mirror list for the $basearch repo.

mirrors/debuginfo

Contains a mirror list for the debuginfo repo.

mirrors/SRPMS

Contains a mirror list for the SRPMS repo.

mirrors/testing

Contains a mirror list for the testing/$basearch repo.

This directory structure can be hosted on a website, ftp server, or on the top level of a DVD image.

Note If the context is included in the top level of a DVD it will be automatically scanned and its contents will be listed for selection.

4.1. Context Metadata

In the %post section of yum-conf-context-example the local yum database will be configured to install the yum group example.

4.1.1. Package Groups

The default package set of a Scientific Linux system is often a product of the selected Package Groups. These are defined by the repository’s metadata in a file either named comps.xml or group.xml.

Note The traditional name is comps.xml and will be used here going forward.

Anaconda (the installation program), yum, and dnf, all take the content of every enabled repository into account before determining a course of action.

This behavior is not unexpected.

The default behavior during package installation, for example, is to install the package with the highest version number – regardless of which repository contains the package.

This behavior extends to Package Groups. The group metadata from all available comps.xml files from each repository is merged into one namespace before any packages are selected. As a result, your Context can add packages to the most common groups easily.

Your Context must include a group named after your context that can be utilized for adding your feature set. This group may contain no packages.

Tip You can generate your own comps.xml with the yum-groups-manager tool. 
yum-groups-manager --id example -n example --description='Support file for the example context' --merge comps.xml --mandatory example-base_virt-tools
yum-groups-manager --id example -n example --merge comps.xml --optional example-conf_kerberos
yum-groups-manager --id example -n example --merge comps.xml example-logo_mycompany

4.2. Generating Your Update Information

Over time your Context will probably have package updates. Each updated package should include information about the update.

Scientific Linux has standardized on the upstream updateinfo.xml metadata for this information.

Tip Scientific Linux provides a full python library for generating and manipulating updateinfo.xml documents within the python-Updateinfo package contained in the example context.
Tutorial information for python-Updateinfo is maintained with that library.

Scientific Linux hosted Contexts must have complete update information for all packages within the production repositories.

Note Packages within the testing repositories are strongly encouraged to provide this metadata, but it is not required.

4.3. Naming Your RPM Packages

The naming convention for Context packages is designed to produce clear origin of packages and quick understanding of their purposes.

The format of context packages consists of:

In this way you can quickly filter packages by Context, behavior, and purpose.

Examples:

  • example-base_virt-tools

  • example-conf_kerberos

  • example-logo_mycompany

  • example-util_krb5-appl-clients

Note Well known packages with defined names(such as python support libraries or commonly used applications such as rootd) are not required to rename their packages.
However, they are encouraged to add a rpm Provides for what the package would be named if it were renamed for the context.

5. Submitting Your Context to Scientific Linux

TBA

6. Code

Currently the code is accessible at:

https://cdcvs.fnal.gov/redmine/projects/scientificlinux/repository/anaconda-context
git clone https://cdcvs.fnal.gov/projects/scientificlinux-anaconda-context