{"id":10404,"date":"2024-07-30T10:48:08","date_gmt":"2024-07-30T08:48:08","guid":{"rendered":"https:\/\/cms.zdv.uni-mainz.de\/forschungsdaten\/?page_id=10404"},"modified":"2025-02-04T11:50:47","modified_gmt":"2025-02-04T10:50:47","slug":"irods-wiki","status":"publish","type":"page","link":"https:\/\/www.forschungsdaten.uni-mainz.de\/en\/irods-wiki\/","title":{"rendered":"iRODS Wiki"},"content":{"rendered":"<\/jgu-base-heading>\n\n\n\n \n<\/jgu-base-anchornavigation>\n\n<\/div>\n\n\n\n

<\/p>\n\n\n<\/jgu-base-heading>\n\n\n\n

iRODS is a decentralized virtual file system that is separate from your normal directories in Linux or Windows.\nIn iRODS, directories are called collections and they can contain further sub-collections (sub-directories) or data objects (files).\nIn practice, you can treat collections and data as directories and files for the sake of simplicity.\nFiles and folders in your own local and mounted directories can be archived (copied) to the remote iRODS file system provided by the ZDV.\nArchiving is done by first creating a directory (collection) in the remote path and then copying your local files and directories to the remote path in iRODS.\nThe archived files can be copied back or downloaded years later.\nThe archives can be private and only stored long-term, they can be restricted to research groups or offered for public download. <\/p>\n\n\n\n

<\/p>\n\n\n\n


<\/strong><\/p>\n\n\n<\/jgu-base-heading>\n\n\n\n

Archiving in iRODS is currently possible via the command line of the ZDV Linux work PCs, the web server linux.zdv.uni-mainz.de and the supercomputer MOGON II\/MOGON-NHR. MOGON also has a special iRODS wiki<\/a>. In the future, there will be new and more user-friendly options for accessing iRODS. Currently, only users with a university account can use iRODS. If your current work computer does not have a standard ZDV Linux installation, you can access the Linux servers via SSH. From there, you can use the available iRODS iCommands to archive data. Personal computers with an iRODS iCommand client installation can also access the ZDV iRODS archive. <\/p>\n\n\n\n

Before accessing the iRODS server, a configuration file must be available and you must authenticate yourself with your JGU account.\nThe configuration file “irods_environment.json” is located in the hidden folder “.irods” and has the following content: <\/p>\n\n\n\n

{ \n  \"irods_authentication_scheme\": \"pam_password\",\n  \"irods_client_server_negotiation\": \"request_server_negotiation\",\n  \"irods_client_server_policy\": \"CS_NEG_REQUIRE\",\n  \"irods_encryption_algorithm\": \"AES-256-CBC\",\n  \"irods_encryption_key_size\": 32,\n  \"irods_encryption_num_hash_rounds\": 16,\n  \"irods_encryption_salt_size\": 8,\n  \"irods_host\": \"irods.zdv.uni-mainz.de\",\n  \"irods_port\": 1247,\n  \"irods_user_name\": \"FILL IN YOUR USERNAME HERE\",\n  \"irods_home\": \"\/zdv\/home\/FILL IN YOUR USERNAME HERE\",\n  \"irods_zone_name\": \"zdv\"\n}<\/code><\/pre>\n\n\n\n
Do not forget to enter your correct user name.\nThen enter the command “iinit” and your JGU password. <\/p>\n\n\n\n
<\/p>\n\n\n\n
<\/p>\n\n\n<\/jgu-base-heading>\n\n\n\n
\"\"<\/p>\n\n\n\n
<\/p>\n\n\n<\/jgu-base-heading>\n\n\n\n
Here is a short summary of the most important iRODS commands with some important command line parameters.\nNote that they start with an ‘ i ‘: <\/p>\n\n\n\n
Command<\/th>Parameters<\/th>Description<\/th><\/tr>
ipwd<\/code><\/td><\/td>Print current iRODS working directory (collection)<\/td><\/tr>
ils<\/code><\/td>-l, -L, -A, -r<\/td>Lists of the iRODS directory (collection) (-l: with details; -L: more details; -A: ACL, -r recursive)<\/td><\/tr>
icd<\/code><\/td><\/td>Change iRODS directory (collection) to the target path<\/td><\/tr>
imkdir<\/code><\/td>-p  <\/td>create a new directory (collection) (directory; -p: create full path with parents)<\/td><\/tr>
iput<\/code><\/td>-K, -r, –metadata<\/td>Upload files\/directories, (K: calculate and validate checksums; -r: recursive, –metadata add descriptions of the data: “Title;My data;;Description;Research data;;”)<\/td><\/tr>
iget<\/code><\/td>-r, -f  <\/td>Download iRODS target data to the current local path, -r recursive, -f overwrite<\/td><\/tr>
iticket<\/code><\/td>create read  <\/td>Create a ticket for downloading files from the Internet<\/td><\/tr>
imeta<\/code><\/td>ls -d\/-C  <\/td>Listing the metadata of an iRODS file or directory (collection)<\/td><\/tr>
imeta<\/code><\/td>set -d\/-C  Key “Value”<\/td>Add\/update metadata to an iRODS file or directory (collection); e.g: Key = Title, “Value” = My research data: Key = Title, “Value” = My research data<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
<\/p>\n\n\n\n
<\/p>\n\n\n\n
<\/p>\n\n\n<\/jgu-base-heading>\n\n\n\n
Here is a brief summary of our iRODS wrapper commands that simplify the archiving process of research data.\nNote that wrappers start with an ‘ i_ ‘, unlike pure iRODS commands which start with ‘i’: <\/p>\n\n\n\n
Command<\/th>Parameters<\/th>Description<\/th><\/tr>
source \/usr\/local\/bin\/i_init.sh<\/code><\/td>For linux.zdv.uni-mainz.de<\/td>Initializes iRODS and displays the current remote iRODS directory after each subsequent command.<\/td><\/tr>
i_exit<\/code><\/td><\/td>Deletes the current iRODS remote directory and cleans up the local settings.<\/td><\/tr>
i_archive<\/code><\/td> <\/td>Uploads the 1st argument to the current remote iRODS directory\/collection and adds the metadata of the .json file of the second argument to all uploaded files.<\/td><\/tr>
i_metaupdate<\/code><\/td> <\/td>Updates the (first argument) remote iRODS directory\/collection and updates the metadata (second argument) using the .json file.<\/td><\/tr>
i_publish<\/code><\/td><\/td>Creates an iRODS web download ticket and links for all files in the path (first argument).\nAnyone can use the links to download the files. <\/td><\/tr>
i_ticketget<\/code><\/td><\/td>Prints the iRODS ticket of a remote iRODS path, if available.<\/td><\/tr>
i_downlinkget<\/code><\/td><\/td>Uses the iRODS ticket and prints the public web download links.<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
<\/p>\n\n\n\n
A simplified procedure for archiving research data would use the simplified wrappers and look as follows:<\/p>\n\n\n\n
source \/usr\/local\/bin\/i_init.sh for the systems linux.zdv.uni-mainz.de
Provides the wrapper commands and initializes iRODS<\/p>\n\n\n\n
imkdir -p \/zdv\/home\/MUSTERPERSON\/project123\/
creates a new remote iRODS path to which the data is copied;<\/p>\n\n\n\n
icd \/zdv\/home\/MUSTERPERSON\/project123\/
changes the current remote iRODS directory to the newly created one;<\/p>\n\n\n\n
i_archive \/fullpath\/some.file metadata.json
archives a local file some.file into the current remote iRODS path by copying it and transferring the metadata from the metadata.json files.<\/p>\n\n\n\n
i_publish \/zdv\/home\/MUSTERPERSON\/project123\/some.file
creates a ticket number (e.g.: ABCDEFG123456890) and prints the generated web download link.<\/p>\n\n\n\n
<\/p>\n\n\n<\/jgu-base-heading>\n\n\n\n
<\/p>\n\n\n\n
Metadata is accepted by iRODS as triplets (Attribute [i.e: Title], Value [i.e: “Research Data from X publication”],, Unit[meist immer leer gelassen]).
The first two fields Attribute and Value are mandatory fields and must not be empty, the Unit is optional.<\/p>\n\n\n\n
You can manually add metadata to each remote iRODS file and directory after uploading by selecting
imeta set -d some.file Title \"My research data from Publication X\"<\/code><\/p>\n\n\n\n
This would be very complex and time-consuming, so it is advisable to define a .json metadata file for all files and directories of a single archive tree.<\/p>\n\n\n\n
i_archive some.file metadata.json<\/code>
Can be used to upload your local files with metadata all at once.<\/p>\n\n\n\n
i_metaupdate \/zdv\/home\/MUSTERPERSON\/project123\/some.file metadata.json<\/code>
Can be used to update a single remote file\/directory with metadata.<\/p>\n\n\n\n
This flat metadata.json example requires a minimal set of metadata attributes:<\/p>\n\n\n\n
{\n \"Title\":\"\",\n \"ResourceType\":\"\",\n \"Project\":\"\",\n \"Keywords\":\"\",\n}\n<\/code><\/pre>\n\n\n\n
{\n \"Title\":\"My Scientific Data from XYZ publication\",\n \"ResourceType\":\"Tables, Texts, Images\",\n \"Project\":\"BMBF-12345, DFG-67890\",\n \"Keywords\":\"Thermodynamics, Simulation, HPC, MPI , XYZ, ,BMBF, DFG\",\n}\n\n\n<\/code><\/pre>\n\n\n\n
The following attributes are set automatically when uploading files\/directories and are not required in the .json file:<\/p>\n\n\n\n
Creator, Publisher, Location, Date, ExpiryDate(Date + 10 years), protected (default: \"false\") .<\/pre>\n\n\n\n
The following attributes are recommended in your .json metadata file:<\/p>\n\n\n\n
{\n \"Title\":\"\",\n \"ResourceType\":\"\",\n \"Project\":\"\",\n \"Keywords\":\"\",\n \"Contributor\":\"\",\n \"Reference\":\"\",\n \"License\":\"\"\n}\n<\/code><\/pre>\n\n\n\n
{\n \"Title\":\"My Scientific Data from XYZ publication\",\n \"ResourceType\":\"Tables, Texts, Images\",\n \"Project\":\"BMBF-12345, DFG-67890\",\n \"Keywords\":\"Thermodynamics, Simulation, HPC, MPI , XYZ, ,BMBF, DFG\",\n \"Contributor\":\"Co-author1, Co-author2, Co-author3\",\n \"Reference\":\"\",\n \"License\":\"GPLX, CC0, CC-BY\"\n}\n<\/code><\/pre>\n\n\n\n
You can confirm that the metadata has been set by using:
imeta ls -d \/zdv\/home\/MUSTERPERSON\/project123\/some.file<\/code> for files
imeta ls -C \/zdv\/home\/MUSTERPERSON\/project123\/<\/code> for directories.<\/p>\n\n\n\n
Summary of imeta:<\/p>\n\n\n\n
Parameters<\/th>Description<\/th><\/tr><\/thead>
add|set|rm|ls|cp<\/td>command, see the next table for details(ls|cp<\/em> do not need the AVU triplet)<\/td><\/tr>
-d dataObject |-C directory\/collection<\/td>which object\/collection (file\/path) is to be queried\/edited<\/td><\/tr>
Attributes Value  [Unit]<\/td>AVU Triplet, in which the unit<\/em> is optional<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
<\/p>\n\n\n\n
Command description:<\/p>\n\n\n\n
Command<\/th>Description<\/th><\/tr><\/thead>
add<\/td>add an AV(U) triplet<\/td><\/tr>
set<\/td>Set a single value<\/td><\/tr>
rm<\/td>Remove an AV(U) triplet<\/td><\/tr>
ls<\/td>List existing metadata.\nIf attribute is specified, only metadata of the specified attribute is listed <\/td><\/tr>
cp<\/td>Copy existing metadata.\nRequires a destination and a source (e.g. imeta cp -d source -c target<\/code>) <\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
<\/p>\n\n\n\n
Publishing research data:<\/p>\n\n\n\n
A ticket for downloading files must be created for public access.
With this ticket and the path to the remote iRODS storage location, anyone can download the information and content.
Already archived data is published with :<\/p>\n\n\n\n
 i_publish<\/code><\/p>\n\n\n\n
i_publish \/zdv\/home\/MUSTERPERSON\/project123\/some.file<\/code>
generates a ticket number (i.e: ABCDEFG123456890) and prints the generated web download link<\/p>\n\n\n\n
You can also perform the publishing process manually using iRODS commands instead:
iticket create read \/zdv\/home\/MUSTERPERSON\/project123\/some.file<\/code>
generates a ticket number (i.e: ABCDEFG123456890) udie is used to download some.file from the network
Then you need to add the path so that it looks like this: wget https:\/\/irods-web.zdv.uni-mainz.de\/irods-rest\/rest\/fileContents\/zdv\/home\/MUSTERPERSON\/some.file?ticket=ABCDEFG123456890<\/code>Data Policy and Licensing<\/p>\n\n\n\n
The “creator” is responsible for ensuring that the further use of third-party data is lawful (copyright law) and that personal data is handled correctly (GDPR).
This applies to all archived data, even if the “creator” is no longer employed at the university or if the data is not public.
This decision aid can help you to decide whether your data may be published.<\/p>\n\n\n\n
If your data can actually be published, there are different types of licenses for different cases:<\/p>\n\n\n\n