|
<!DOCTYPE html>
|
|
<html lang="en">
|
<head>
|
<meta charset="utf-8" />
|
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="generator" content="Docutils 0.18.1: http://docutils.sourceforge.net/" />
|
|
|
<!-- Licensed under the Apache 2.0 License -->
|
<link rel="stylesheet" type="text/css" href="_static/fonts/open-sans/stylesheet.css" />
|
<!-- Licensed under the SIL Open Font License -->
|
<link rel="stylesheet" type="text/css" href="_static/fonts/source-serif-pro/source-serif-pro.css" />
|
<link rel="stylesheet" type="text/css" href="_static/css/bootstrap.min.css" />
|
<link rel="stylesheet" type="text/css" href="_static/css/bootstrap-theme.min.css" />
|
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
|
<title>Track & Evaluation — MULTI-PARTY MEETING TRANSCRIPTION CHALLENGE 2.0</title>
|
<link rel="stylesheet" type="text/css" href="_static/pygments.css" />
|
<link rel="stylesheet" type="text/css" href="_static/guzzle.css" />
|
<script data-url_root="./" id="documentation_options" src="_static/documentation_options.js"></script>
|
<script src="_static/jquery.js"></script>
|
<script src="_static/underscore.js"></script>
|
<script src="_static/_sphinx_javascript_frameworks_compat.js"></script>
|
<script src="_static/doctools.js"></script>
|
<script src="_static/sphinx_highlight.js"></script>
|
<script async="async" src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js"></script>
|
<link rel="index" title="Index" href="genindex.html" />
|
<link rel="search" title="Search" href="search.html" />
|
<link rel="next" title="Baseline" href="Baseline.html" />
|
<link rel="prev" title="Datasets" href="Dataset.html" />
|
|
|
|
</head><body>
|
<div class="related" role="navigation" aria-label="related navigation">
|
<h3>Navigation</h3>
|
<ul>
|
<li class="right" style="margin-right: 10px">
|
<a href="genindex.html" title="General Index"
|
accesskey="I">index</a></li>
|
<li class="right" >
|
<a href="Baseline.html" title="Baseline"
|
accesskey="N">next</a> |</li>
|
<li class="right" >
|
<a href="Dataset.html" title="Datasets"
|
accesskey="P">previous</a> |</li>
|
<li class="nav-item nav-item-0"><a href="index.html">MULTI-PARTY MEETING TRANSCRIPTION CHALLENGE 2.0</a> »</li>
|
<li class="nav-item nav-item-this"><a href="">Track & Evaluation</a></li>
|
</ul>
|
</div>
|
<div class="container-wrapper">
|
|
<div id="mobile-toggle">
|
<a href="#"><span class="glyphicon glyphicon-align-justify" aria-hidden="true"></span></a>
|
</div>
|
<div id="left-column">
|
<div class="sphinxsidebar"><a href="
|
index.html" class="text-logo">MULTI-PARTY MEETING TRANSCRIPTION CHALLENGE 2.0</a>
|
<div class="sidebar-block">
|
<div class="sidebar-wrapper">
|
<div id="main-search">
|
<form class="form-inline" action="search.html" method="GET" role="form">
|
<div class="input-group">
|
<input name="q" type="text" class="form-control" placeholder="Search...">
|
</div>
|
<input type="hidden" name="check_keywords" value="yes" />
|
<input type="hidden" name="area" value="default" />
|
</form>
|
</div>
|
</div>
|
</div>
|
<div class="sidebar-block">
|
<div class="sidebar-toc">
|
|
|
<p class="caption" role="heading"><span class="caption-text">Contents:</span></p>
|
<ul class="current">
|
<li class="toctree-l1"><a class="reference internal" href="Introduction.html">Introduction</a><ul>
|
<li class="toctree-l2"><a class="reference internal" href="Introduction.html#call-for-participation">Call for participation</a></li>
|
<li class="toctree-l2"><a class="reference internal" href="Introduction.html#timeline-aoe-time">Timeline(AOE Time)</a></li>
|
<li class="toctree-l2"><a class="reference internal" href="Introduction.html#guidelines">Guidelines</a></li>
|
</ul>
|
</li>
|
<li class="toctree-l1"><a class="reference internal" href="Dataset.html">Datasets</a><ul>
|
<li class="toctree-l2"><a class="reference internal" href="Dataset.html#overview-of-training-data">Overview of training data</a></li>
|
<li class="toctree-l2"><a class="reference internal" href="Dataset.html#detail-of-alimeeting-corpus">Detail of AliMeeting corpus</a></li>
|
<li class="toctree-l2"><a class="reference internal" href="Dataset.html#get-the-data">Get the data</a></li>
|
</ul>
|
</li>
|
<li class="toctree-l1 current"><a class="current reference internal" href="#">Track & Evaluation</a><ul>
|
<li class="toctree-l2"><a class="reference internal" href="#speaker-attributed-asr">Speaker-Attributed ASR</a></li>
|
<li class="toctree-l2"><a class="reference internal" href="#evaluation-metric">Evaluation metric</a></li>
|
<li class="toctree-l2"><a class="reference internal" href="#sub-track-arrangement">Sub-track arrangement</a></li>
|
</ul>
|
</li>
|
<li class="toctree-l1"><a class="reference internal" href="Baseline.html">Baseline</a><ul>
|
<li class="toctree-l2"><a class="reference internal" href="Baseline.html#overview">Overview</a></li>
|
<li class="toctree-l2"><a class="reference internal" href="Baseline.html#quick-start">Quick start</a></li>
|
<li class="toctree-l2"><a class="reference internal" href="Baseline.html#baseline-results">Baseline results</a></li>
|
</ul>
|
</li>
|
<li class="toctree-l1"><a class="reference internal" href="Rules.html">Rules</a></li>
|
<li class="toctree-l1"><a class="reference internal" href="Challenge_result.html">Challenge Result</a></li>
|
<li class="toctree-l1"><a class="reference internal" href="Organizers.html">Organizers</a></li>
|
<li class="toctree-l1"><a class="reference internal" href="Contact.html">Contact</a></li>
|
</ul>
|
|
|
</div>
|
</div>
|
|
</div>
|
</div>
|
<div id="right-column">
|
|
<div role="navigation" aria-label="breadcrumbs navigation">
|
<ol class="breadcrumb">
|
<li><a href="index.html">Docs</a></li>
|
|
<li>Track & Evaluation</li>
|
</ol>
|
</div>
|
|
<div class="document clearer body">
|
|
<section id="track-evaluation">
|
<h1>Track & Evaluation<a class="headerlink" href="#track-evaluation" title="Permalink to this heading">¶</a></h1>
|
<section id="speaker-attributed-asr">
|
<h2>Speaker-Attributed ASR<a class="headerlink" href="#speaker-attributed-asr" title="Permalink to this heading">¶</a></h2>
|
<p>The speaker-attributed ASR task poses a unique challenge of transcribing speech from multiple speakers and assigning a speaker label to the transcription. Figure 2 illustrates the difference between the speaker-attributed ASR task and the multi-speaker ASR task. This track allows for the use of the AliMeeting, Aishell4, and Cn-Celeb datasets as constrained data sources during both training and evaluation. The AliMeeting dataset, which was used in the M2MeT challenge, includes Train, Eval, and Test sets. Additionally, a new Test-2023 set, consisting of approximately 10 hours of meeting data recorded in an identical acoustic setting as the AliMeeting corpus, will be released soon for challenge scoring and ranking. It’s worth noting that the organizers will not provide the near-field audio, transcriptions, or oracle timestamps of the Test-2023 set. Instead, segments containing multiple speakers will be provided, which can be obtained using a simple voice activity detection (VAD) model.</p>
|
<p><img alt="task difference" src="_images/task_diff.png" /></p>
|
</section>
|
<section id="evaluation-metric">
|
<h2>Evaluation metric<a class="headerlink" href="#evaluation-metric" title="Permalink to this heading">¶</a></h2>
|
<p>The accuracy of a speaker-attributed ASR system is evaluated using the concatenated minimum permutation character error rate (cpCER) metric. The calculation of cpCER involves three steps. Firstly, the reference and hypothesis transcriptions from each speaker in a session are concatenated in chronological order. Secondly, the character error rate (CER) is calculated between the concatenated reference and hypothesis transcriptions, and this process is repeated for all possible speaker permutations. Finally, the permutation with the lowest CER is selected as the cpCER for that session. TThe CER is obtained by dividing the total number of insertions (Ins), substitutions (Sub), and deletions(Del) of characters required to transform the ASR output into the reference transcript by the total number of characters in the reference transcript. Specifically, CER is calculated by:</p>
|
<div class="math notranslate nohighlight">
|
\[\text{CER} = \frac {\mathcal N_{\text{Ins}} + \mathcal N_{\text{Sub}} + \mathcal N_{\text{Del}} }{\mathcal N_{\text{Total}}} \times 100\%,\]</div>
|
<p>where <span class="math notranslate nohighlight">\(\mathcal N_{\text{Ins}}\)</span>, <span class="math notranslate nohighlight">\(\mathcal N_{\text{Sub}}\)</span>, <span class="math notranslate nohighlight">\(\mathcal N_{\text{Del}}\)</span> are the character number of the three errors, and <span class="math notranslate nohighlight">\(\mathcal N_{\text{Total}}\)</span> is the total number of characters.</p>
|
</section>
|
<section id="sub-track-arrangement">
|
<h2>Sub-track arrangement<a class="headerlink" href="#sub-track-arrangement" title="Permalink to this heading">¶</a></h2>
|
<section id="sub-track-i-fixed-training-condition">
|
<h3>Sub-track I (Fixed Training Condition):<a class="headerlink" href="#sub-track-i-fixed-training-condition" title="Permalink to this heading">¶</a></h3>
|
<p>Participants are required to use only the fixed-constrained data (i.e., AliMeeting, Aishell-4, and CN-Celeb) for system development. The usage of any additional data is strictly prohibited. However, participants can use open-source pre-trained models from third-party sources, such as <a class="reference external" href="https://huggingface.co/models">Hugging Face</a> and <a class="reference external" href="https://www.modelscope.cn/models">ModelScope</a>, provided that the list of utilized models is clearly stated in the final system description paper.</p>
|
</section>
|
<section id="sub-track-ii-open-training-condition">
|
<h3>Sub-track II (Open Training Condition):<a class="headerlink" href="#sub-track-ii-open-training-condition" title="Permalink to this heading">¶</a></h3>
|
<p>Participants are allowed to use any publicly available data set, privately recorded data, and manual simulation to build their systems in addition to the fixed-constrained data. They can also utilize any open-source pre-trained models, but it is mandatory to provide a clear list of the data and models used in the final system description paper. If manually simulated data is used, a detailed description of the data simulation scheme must be provided.</p>
|
</section>
|
</section>
|
</section>
|
|
|
</div>
|
|
<div class="footer-relations">
|
|
<div class="pull-left">
|
<a class="btn btn-default" href="Dataset.html" title="previous chapter (use the left arrow)">Datasets</a>
|
</div>
|
|
<div class="pull-right">
|
<a class="btn btn-default" href="Baseline.html" title="next chapter (use the right arrow)">Baseline</a>
|
</div>
|
</div>
|
<div class="clearer"></div>
|
|
</div>
|
<div class="clearfix"></div>
|
</div>
|
<div class="related" role="navigation" aria-label="related navigation">
|
<h3>Navigation</h3>
|
<ul>
|
<li class="right" style="margin-right: 10px">
|
<a href="genindex.html" title="General Index"
|
>index</a></li>
|
<li class="right" >
|
<a href="Baseline.html" title="Baseline"
|
>next</a> |</li>
|
<li class="right" >
|
<a href="Dataset.html" title="Datasets"
|
>previous</a> |</li>
|
<li class="nav-item nav-item-0"><a href="index.html">MULTI-PARTY MEETING TRANSCRIPTION CHALLENGE 2.0</a> »</li>
|
<li class="nav-item nav-item-this"><a href="">Track & Evaluation</a></li>
|
</ul>
|
</div>
|
<script type="text/javascript">
|
$("#mobile-toggle a").click(function () {
|
$("#left-column").toggle();
|
});
|
</script>
|
<script type="text/javascript" src="_static/js/bootstrap.js"></script>
|
<div class="footer">
|
© Copyright 2023, Speech Lab, Alibaba Group; ASLP Group, Northwestern Polytechnical University. Created using <a href="http://sphinx.pocoo.org/">Sphinx</a>.
|
</div>
|
</body>
|
</html>
|
