LOLA11 Banner
Join

Details

Last modified on April 22, 2021.

Goal of the study

The goal of LOLA11 is to evaluate the performance of state-of-the-art lung and lobe segmentation methods for chest CT scans.  Many algorithms for lung and lobe segmentation have been published, but directly comparing them is difficult; different methods are usually evaluated on different datasets using different evaluation measures. Some methods can be re-implemented based on the available publication, but there are often parameters to be set for which in-depth knowledge of the method is required or training data is needed that is not publicly available. LOLA11 provides a data set of chest CT scans with varying abnormalities for which reference standards of lung and lobe segmentations have been established. By comparing all methods against this same reference standard, a fair and meaningful evaluation is carried out. LOLA11 thus provides a unique opportunity to compare lung and lobe segmentation algorithms.

A summary article based on the initial phase of the study was meant to be published, but unfortunately this paper was never finished.  However, the LOLA11 website was ported to grand-challenge.org and keeps running using automated evaluation. Teams are encouraged to submit improved results. Teams which did not participate in the initial phase are always welcome to register and submit their results. In this way, the LOLA11 website will continue to contribute an overview of the state-of-the-art in lung and lobe segmentation in chest CT scans.

Task

Given a volumetric chest CT scan, the task is to segment the lungs and/or the lobes, that is, to label each voxel as one of six classes:

  •  (0)  -  outside the lungs
  • (10) -  left lung, upper lobe
  • (11) -  left lung, lower lobe
  • (20) -  right lung, upper lobe
  • (21) -  right lung, middle lobe
  • (22) -  right lung, lower lobe.

We also allow a simpler labeling, using three classes:

  • (0) -  outside the lungs
  • (1) -  left lung
  • (2) -  right lung

Results are published for these two types of algorithms separately:

  • Automatic: scan is the only input
  • Semi-automatic: Some user input is required. E.g. Different scans use different manually set algorithm parameters or manually set seedpoints are used.

Dates

2011:

  • June 22:  Teams who have submitted results to LOLA11 before this date now receive their results table. Teams that submit results after this date (results can be submitted until the paper submission deadline) receive their result table immediately.
  • July 8: Paper submission deadline for LOLA11 papers.
  • July 15: Notification of acceptance.
  • July 27: Final camera-ready papers due.
  • September 18: The Fourth International Workshop on Pulmonary Image Analysis
    The proceedings are available on Amazon.

Rules

The collection of the data, the organization of the LOLA11 study, and the maintenance of this website require a large effort. In the spirit of cooperative scientific progress, we are committed to maintaining this site as a public repository of benchmark results for lung and lobe segmentation. In return, we ask everyone who uses this site to respect the rules below.

These rules amount to a simple tit for tat: we actively encourage anyone to use this data for testing lung and lobe segmentation algorithms. In return, we ask you to send us the results of your method and a document that describes your method. The score of your algorithm and your description will be publicly available on this site.

We do not claim any ownership or rights to the algorithms or uploaded documents, and do not want to create any obstacles for publishing methods that use the LOLA11 data.

The following rules apply to those who register a team and download the data:

  • The downloaded data or any data derived from these data may not be given or redistributed under any circumstances to persons not belonging to the registered team.
  • All information entered when registering a team, including the name of the contact person, the affiliation (institute, organization, or company the team's contact person works for) and the e-mail address must be complete and correct. In other words, anonymous registration is not allowed. 
  • Data downloaded from this site may only be used for the purpose of preparing an entry to be submitted on this site. The data may not be used for other purposes in scientific studies and may not be used to train or develop other algorithms, including but not limited to algorithms used in commercial products.
  • Each result submitted on this site must be accompanied by a pdf file describing the details of the system. The LOLA11 organizers reserve the right to refuse to evaluate systems whose description does not meet minimal requirements. More explanation about the required description is given below.
  • Results uploaded to this website will be made publicly available on this site. By submitting results, you grant us permission to do so. Teams maintain full ownership and rights to the method.
  • Teams are required to notify the maintainers of this site about any publication that is (partly) based on the data on this site, in order for us to maintain a list of publications associated with the LOLA11 study.

Data

For this study, a number of chest CT scans is available for download. The scans come from a variety of sources and represent a variety of clinically common scanners and protocols. The scans have been selected such that in approximately half of the scans lung and/or lobe segmentation is deemed 'easy' and in the other half 'hard. The maximum slice spacing present is 1.5mm, where most scans are (near) isotropic. To ensure consistent evaluation, reference segmentations for the data cannot be downloaded and will not be made available in the future.

The division of the cases into 'easy' and 'hard' cases is done by the organizers. The definition of easy and hard could be based on the extent of the abnormalities present, however, the difficulty for segmentation tasks is not always easy to define that way, e.g., scans in which severe emphysema is present are usually easy for lung segmentation algorithms despite a large amount of abnormal lung tissue present. Therefore, we chose to define easy and hard based on the results of the submitted algorithms for the workshop: by overlaying all results, areas with little overlap between the methods can be defined as hard to segment, whereas regions where all methods agree are easy to segment. Based on this, scans are categorized as easy or hard for lung and/or lobe segmentation. Note that scans can be easy for lung segmentation and hard for lobe segmentation at the same time (and vice versa).

There is no separate data set available to train algorithms. Of course, you are allowed to use your own training data for any algorithm you develop and apply to the LOLA11 test data set. However, you should not use the LOLA11 data in any way for training your system as this would positively bias and thus invalidate your results. We have no way of checking if teams use the (results on) test data to train, tune or tweak their performance. We ask you to not do this.

Data format

Originally, files were distributed in tar.bz2 archives. In 2021, we have converted all 55 scans to losslessly compressed .mha and make them available on Zenodo.

Each downloaded file contains original CT scans, stored in Meta format and can be read with it ITK.  An application that can read the data is SNAP. 

Reference standard protocol

The reference standard was produced by manual segmentation of the lungs and lobes. A set of rules was used to indicate where to let the segmentation of the lungs intersect the pulmonary vessels and bronchi in the hilar region. Parts of the bronchial and pulmonary vessel tree distal from these intersections are considered to be a part of the lung volume. The rules used are detailed below, numbers between brackets refer to segmental bronchial branches.

right lung:

upper lobe: immediately before the first division of the right upper lobe bronchus
middle lobe: immediately before the first division of the right middle lobe bronchus (division into 4/5)
lower lobe: immediately before the division of the right lower lobe bronchus into the basal segmental bronchi (division into 7/8/9/10). Note that the apical or superior segmental bronchus of the right lower lobe (6) is to be included in the lung volume.

left lung:

upper lobe: immediately before the first division of the left upper lobe bronchus
lower lobe/lingula: immediately before the first division of the left lower lobe lingular bronchus (division into 4/5)
lower lobe: immediately before the division of the left lower lobe bronchus into the basal segmental bronchi (division into 7/8/9/10). Note that the apical or superior segmental bronchus of the left lower lobe (6) is to be included in the lung volume.

Evaluation protocol

For the workshop, the evaluation protocol will be kept simple: overall overlap measures for each lung and lobe. To avoid problems with the orientation of the scans, we will evaluate each scan both in the original orientation and upside down and use the best result. After the workshop, an overview paper with all participants will be prepared that will contain a more elaborate evaluation. This will be discussed at the workshop.

The overlap between two binary segmentation volumes (e.g. left upper lobe in reference standard and result) is defined as the volume of their intersection divided by the volume of their union. Since exactly drawing the lung and lobar borders manually is sometimes difficult and subjective (e.g. in case of abnormalities at the lung border or incomplete fissures), a slack border is defined as all voxels within 2mm of a manually drawn border in the reference standard. Voxels in this slack border are not taken into account during evaluation. The size of the slack border was determined based on interobserver agreement for manually contouring the lobar borders in the 55 cases. The mean distance between the borders drawn by two human observers was 1.50mm, with a standard deviation of 1.28mm. The median distance was 1.08mm. 

For the workshop, the results will be reported in terms of mean, standard deviation, minimum, first quartile, median, third quartile, and maximum overlap over the 55 scans for each object separately. The overall score for a lung or lobe segmentation is simply the mean of the means.

Submitting results

  • The results file you submit should be a single, compressed file (archive) as a .zip as this is the most portable between platforms.
  • The archive will be decompressed to a single folder and should hold 55 results files. The 55 files will be processed in alphabetical order and the first file is assumed to be case 1 and the last case 55.
  • For your convenience, we provide the byte sizes for each case:

case 1: 121634816
case 2: 121634816
case 3: 110624768
case 4: 69206016
case 5: 93847552
case 6: 84410368
case 7: 254541824
case 8: 157548544
case 9: 118226944
case 10: 112197632
case 11: 102236160
case 12: 120324096
case 13: 168296448
case 14: 74973184
case 15: 88604672
case 16: 78905344
case 17: 137363456
case 18: 107741184
case 19: 92798976
case 20: 77332480
case 21: 115343360
case 22: 111149056
case 23: 86507520
case 24: 84672512
case 25: 85721088
case 26: 81788928
case 27: 197394432
case 28: 181665792
case 29: 109576192
case 30: 107741184
case 31: 89915392
case 32: 66322432
case 33: 108527616
case 34: 73138176
case 35: 44040192
case 36: 206307328
case 37: 61341696
case 38: 83361792
case 39: 130285568
case 40: 134217728
case 41: 148111360
case 42: 72876032
case 43: 131858432
case 44: 136052736
case 45: 65798144
case 46: 62652416
case 47: 139984896
case 48: 113508352
case 49: 125304832
case 50: 67371008
case 51: 202637312
case 52: 154140672
case 53: 88866816
case 54: 78118912
case 55: 81788928

  • The description file for each submission is checked by one of the organizers.
  • Several scans in the original lola11 dataset were upside down. This has been corrected in the Zenodo release of the data (https://doi.org/10.5281/zenodo.4708800). To avoid issues with results that have the wrong orientation, we evaluate each scan as submitted but we also evaluate the scan flipped in the caudal-cranial direction. The best evaluation result for each scan is then used.

Description of the algorithm - checklist

Each submission should contain a description file in pdf format. 

For convenience, we provide a checklist below of items that we believe should be mentioned in a description of a lung and lobe segmentation algorithm.

  • Give the overall structure of the algorithm. Does your algorithm search for boundaries or fissures? Does it use airway or other information? Does it use an atlas and/or region growing?
  • Briefly describe each step in the structure of the algorithm (If applicable, which type of algorithms were used for preprocessing? How are different types of information combined?).
  • List limitations of the algorithm. Is the algorithm specifically designed to segment only certain types of scans? Is your algorithm intended for segmenting pathological lungs? Was it optimized to work for scans with thick or thin slices, are other technical scan parameters expected to influence segmentation performance?
  • Was the algorithm trained with example data? If so, describe the characteristics of the training data.
  • If the algorithm has been tested on other databases, you could consider including those results.
  • What is the average runtime of your algorithm, and on which system is this runtime achieved?
  • Is your algorithm automatic or semi-automatic? If user input is used, how much is needed and in what way?