{ "nbformat": 4, "nbformat_minor": 0, "metadata": { "colab": { "provenance": [], "gpuType": "T4" }, "kernelspec": { "name": "python3", "display_name": "Python 3" }, "language_info": { "name": "python" }, "accelerator": "GPU", "gpuClass": "standard" }, "cells": [ { "cell_type": "markdown", "source": [ "# Running MMS-ASR inference in Colab" ], "metadata": { "id": "Rhm7khm6GskV" } }, { "cell_type": "markdown", "source": [ "In this notebook, we will give an example on how to run simple ASR inference using MMS ASR model. \n", "\n", "Credit to epk2112 [(github)](https://github.com/epk2112/fairseq_meta_mms_Google_Colab_implementation)" ], "metadata": { "id": "83HXBIFeJzR8" } }, { "cell_type": "markdown", "source": [ "## Step 1: Clone fairseq-py and install latest version" ], "metadata": { "id": "2GfxksHDGyJv" } }, { "cell_type": "code", "execution_count": 7, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "Cj2x80SegRzr", "outputId": "00f9f833-3ff1-4736-e170-136875b88299" }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "fatal: destination path 'fairseq' already exists and is not an empty directory.\n", "/content/fairseq\n", "/content/fairseq\n", "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n", "Obtaining file:///content/fairseq\n", " Installing build dependencies ... \u001b[?25l\u001b[?25hcanceled\u001b[31mERROR: Operation cancelled by user\u001b[0m\u001b[31m\n", "\u001b[0mLooking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n", "Requirement already satisfied: tensorboardX in /usr/local/lib/python3.10/dist-packages (2.6)\n", "Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from tensorboardX) (1.22.4)\n", "Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from tensorboardX) (23.1)\n", "Requirement already satisfied: protobuf<4,>=3.8.0 in /usr/local/lib/python3.10/dist-packages (from tensorboardX) (3.20.3)\n" ] } ], "source": [ "!mkdir \"temp_dir\"\n", "!git clone https://github.com/pytorch/fairseq\n", "\n", "# Change current working directory\n", "!pwd\n", "%cd \"/content/fairseq\"\n", "!pip install --editable ./ \n", "!pip install tensorboardX\n" ] }, { "cell_type": "markdown", "source": [ "## 2. Download MMS model\n", "Un-comment to download your preferred model.\n", "In this example, we use MMS-FL102 for demo purposes.\n", "For better model quality and language coverage, user can use MMS-1B-ALL model instead (but it would require more RAM, so please use Colab-Pro instead of Colab-Free).\n" ], "metadata": { "id": "cyk4JvZOHSw3" } }, { "cell_type": "code", "source": [ "# MMS-1B:FL102 model - 102 Languages - FLEURS Dataset\n", "!wget -P ./models_new 'https://dl.fbaipublicfiles.com/mms/asr/mms1b_fl102.pt'\n", "\n", "# # MMS-1B:L1107 - 1107 Languages - MMS-lab Dataset\n", "# !wget -P ./models_new 'https://dl.fbaipublicfiles.com/mms/asr/mms1b_l1107.pt'\n", "\n", "# # MMS-1B-all - 1162 Languages - MMS-lab + FLEURS + CV + VP + MLS\n", "# !wget -P ./models_new 'https://dl.fbaipublicfiles.com/mms/asr/mms1b_all.pt'" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "3uZ9WG85gZId", "outputId": "3b13f908-aa8a-4207-9147-60c443ced571" }, "execution_count": 2, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "--2023-05-25 23:53:33-- https://dl.fbaipublicfiles.com/mms/asr/mms1b_fl102.pt\n", "Resolving dl.fbaipublicfiles.com (dl.fbaipublicfiles.com)... 13.227.219.33, 13.227.219.59, 13.227.219.70, ...\n", "Connecting to dl.fbaipublicfiles.com (dl.fbaipublicfiles.com)|13.227.219.33|:443... connected.\n", "HTTP request sent, awaiting response... 200 OK\n", "Length: 4851043301 (4.5G) [binary/octet-stream]\n", "Saving to: ‘./models_new/mms1b_fl102.pt’\n", "\n", "mms1b_fl102.pt 100%[===================>] 4.52G 242MB/s in 20s \n", "\n", "2023-05-25 23:53:53 (230 MB/s) - ‘./models_new/mms1b_fl102.pt’ saved [4851043301/4851043301]\n", "\n" ] } ] }, { "cell_type": "markdown", "source": [ "## 3. Prepare audio file\n", "Create a folder on path '/content/audio_samples/' and upload your .wav audio files that you need to transcribe e.g. '/content/audio_samples/audio.wav' \n", "\n", "Note: You need to make sure that the audio data you are using has a sample rate of 16kHz You can easily do this with FFMPEG like the example below that converts .mp3 file to .wav and fixing the audio sample rate\n", "\n", "Here, we use a FLEURS english MP3 audio for the example." ], "metadata": { "id": "3p5-TQvKHXjO" } }, { "cell_type": "code", "source": [ "!wget -P ./audio_samples/ 'https://datasets-server.huggingface.co/assets/google/fleurs/--/en_us/train/0/audio/audio.mp3'\n", "!ffmpeg -y -i ./audio_samples/audio.mp3 -ar 16000 ./audio_samples/audio.wav" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "cnim4bokprbB", "outputId": "f63ed14f-0bde-4517-ee7b-200ddcc45e5f" }, "execution_count": 3, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "--2023-05-25 23:53:53-- https://datasets-server.huggingface.co/assets/google/fleurs/--/en_us/train/0/audio/audio.mp3\n", "Resolving datasets-server.huggingface.co (datasets-server.huggingface.co)... 50.17.173.235, 44.197.252.161, 3.216.183.114, ...\n", "Connecting to datasets-server.huggingface.co (datasets-server.huggingface.co)|50.17.173.235|:443... connected.\n", "HTTP request sent, awaiting response... 200 OK\n", "Length: 20853 (20K) [audio/mpeg]\n", "Saving to: ‘./audio_samples/audio.mp3’\n", "\n", "audio.mp3 100%[===================>] 20.36K --.-KB/s in 0.09s \n", "\n", "2023-05-25 23:53:53 (238 KB/s) - ‘./audio_samples/audio.mp3’ saved [20853/20853]\n", "\n", "ffmpeg version 4.2.7-0ubuntu0.1 Copyright (c) 2000-2022 the FFmpeg developers\n", " built with gcc 9 (Ubuntu 9.4.0-1ubuntu1~20.04.1)\n", " configuration: --prefix=/usr --extra-version=0ubuntu0.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared\n", " libavutil 56. 31.100 / 56. 31.100\n", " libavcodec 58. 54.100 / 58. 54.100\n", " libavformat 58. 29.100 / 58. 29.100\n", " libavdevice 58. 8.100 / 58. 8.100\n", " libavfilter 7. 57.100 / 7. 57.100\n", " libavresample 4. 0. 0 / 4. 0. 0\n", " libswscale 5. 5.100 / 5. 5.100\n", " libswresample 3. 5.100 / 3. 5.100\n", " libpostproc 55. 5.100 / 55. 5.100\n", "Input #0, mp3, from './audio_samples/audio.mp3':\n", " Metadata:\n", " encoder : Lavf58.45.100\n", " Duration: 00:00:06.88, start: 0.069063, bitrate: 24 kb/s\n", " Stream #0:0: Audio: mp3, 16000 Hz, mono, fltp, 24 kb/s\n", "Stream mapping:\n", " Stream #0:0 -> #0:0 (mp3 (mp3float) -> pcm_s16le (native))\n", "Press [q] to stop, [?] for help\n", "Output #0, wav, to './audio_samples/audio.wav':\n", " Metadata:\n", " ISFT : Lavf58.29.100\n", " Stream #0:0: Audio: pcm_s16le ([1][0][0][0] / 0x0001), 16000 Hz, mono, s16, 256 kb/s\n", " Metadata:\n", " encoder : Lavc58.54.100 pcm_s16le\n", "size= 213kB time=00:00:06.80 bitrate= 256.1kbits/s speed= 398x \n", "video:0kB audio:212kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.035846%\n" ] } ] }, { "cell_type": "markdown", "source": [ "# 4: Run Inference and transcribe your audio(s)\n" ], "metadata": { "id": "44UvHjmMI28Z" } }, { "cell_type": "markdown", "source": [ "In the below example, we will transcribe a sentence in English.\n", "\n", "To transcribe other languages: \n", "1. Go to [MMS README ASR section](https://github.com/facebookresearch/fairseq/tree/main/examples/mms#asr)\n", "2. Open Supported languages link\n", "3. Find your target languages based on Language Name column\n", "4. Copy the corresponding Iso Code\n", "5. Replace `--lang \"eng\"` with new Iso Code\n", "\n", "To improve the transcription quality, user can use language-model (LM) decoding by following this instruction [ASR LM decoding](https://github.com/facebookresearch/fairseq/tree/main/examples/mms#asr)" ], "metadata": { "id": "82Xpxot2wFid" } }, { "cell_type": "code", "source": [ "import os\n", "\n", "os.environ[\"TMPDIR\"] = '/content/temp_dir'\n", "os.environ[\"PYTHONPATH\"] = \".\"\n", "os.environ[\"PREFIX\"] = \"INFER\"\n", "os.environ[\"HYDRA_FULL_ERROR\"] = \"1\"\n", "os.environ[\"USER\"] = \"micro\"\n", "\n", "!python examples/mms/asr/infer/mms_infer.py --model \"/content/fairseq/models_new/mms1b_fl102.pt\" --lang \"eng\" --audio \"/content/fairseq/audio_samples/audio.wav\"\n" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "J8N1RKtBiw5V", "outputId": "db5d3575-bd21-470d-bb4d-e080e8cece50" }, "execution_count": 4, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ ">>> preparing tmp manifest dir ...\n", ">>> loading model & running inference ...\n", "2023-05-25 23:54:02.330426: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.\n", "To enable the following instructions: AVX2 AVX512F FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.\n", "2023-05-25 23:54:04.144981: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n", "===============\n", "Input: /content/fairseq/audio_samples/audio.wav\n", "Output: a tornado is a spinning colum of very low-pressure air which sucks it surrounding air inward and upward\n" ] } ] }, { "cell_type": "markdown", "source": [ "# 5: Beam search decoding using a Language Model and transcribe audio file(s)\n" ], "metadata": { "id": "0j2t8MI4WBiy" } }, { "cell_type": "markdown", "source": [ "Since MMS is a CTC model, we can further improve the accuracy by running beam search decoding using a language model. \n", "\n", "While we have not open sourced the language models used in MMS (yet!), we have provided the details of the data and commands to used to train the LMs in the Appendix section of our paper.\n", "\n", "\n", "For this tutorial, we will use a alternate English language model based on Common Crawl data which has been made publicly available through the efforts of [Likhomanenko, Tatiana, et al. \"Rethinking evaluation in asr: Are our models robust enough?.\"](https://arxiv.org/abs/2010.11745). The language model can be accessed from the GitHub repository [here](https://github.com/flashlight/wav2letter/tree/main/recipes/rasr). " ], "metadata": { "id": "Yfigq1nIWJR6" } }, { "cell_type": "code", "source": [ "! mkdir -p /content/lmdecode \n", "\n", "!wget -P /content/lmdecode https://dl.fbaipublicfiles.com/wav2letter/rasr/tutorial/lm_common_crawl_small_4gram_prun0-6-15_200kvocab.bin # smaller LM \n", "!wget -P /content/lmdecode https://dl.fbaipublicfiles.com/wav2letter/rasr/tutorial/lexicon.txt " ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "sNTd1OCuV8jG", "outputId": "396583b9-400f-4b68-aa20-898f0194da6e" }, "execution_count": 10, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "--2023-05-26 00:16:03-- https://dl.fbaipublicfiles.com/wav2letter/rasr/tutorial/lm_common_crawl_small_4gram_prun0-6-15_200kvocab.bin\n", "Resolving dl.fbaipublicfiles.com (dl.fbaipublicfiles.com)... 13.227.219.33, 13.227.219.70, 13.227.219.10, ...\n", "Connecting to dl.fbaipublicfiles.com (dl.fbaipublicfiles.com)|13.227.219.33|:443... connected.\n", "HTTP request sent, awaiting response... 200 OK\n", "Length: 2627163608 (2.4G) [application/octet-stream]\n", "Saving to: ‘/content/lmdecode/lm_common_crawl_small_4gram_prun0-6-15_200kvocab.bin’\n", "\n", "lm_common_crawl_sma 100%[===================>] 2.45G 27.5MB/s in 94s \n", "\n", "2023-05-26 00:17:37 (26.8 MB/s) - ‘/content/lmdecode/lm_common_crawl_small_4gram_prun0-6-15_200kvocab.bin’ saved [2627163608/2627163608]\n", "\n", "--2023-05-26 00:17:37-- https://dl.fbaipublicfiles.com/wav2letter/rasr/tutorial/lexicon.txt\n", "Resolving dl.fbaipublicfiles.com (dl.fbaipublicfiles.com)... 13.227.219.33, 13.227.219.10, 13.227.219.70, ...\n", "Connecting to dl.fbaipublicfiles.com (dl.fbaipublicfiles.com)|13.227.219.33|:443... connected.\n", "HTTP request sent, awaiting response... 200 OK\n", "Length: 4965720 (4.7M) [text/plain]\n", "Saving to: ‘/content/lmdecode/lexicon.txt’\n", "\n", "lexicon.txt 100%[===================>] 4.74M 5.02MB/s in 0.9s \n", "\n", "2023-05-26 00:17:39 (5.02 MB/s) - ‘/content/lmdecode/lexicon.txt’ saved [4965720/4965720]\n", "\n" ] } ] }, { "cell_type": "markdown", "source": [ "\n", "Install decoder bindings from [flashlight](https://github.com/flashlight/flashlight)\n" ], "metadata": { "id": "8HKmVaRfhHIk" } }, { "cell_type": "code", "source": [ "# Taken from https://github.com/flashlight/flashlight/blob/main/scripts/colab/colab_install_deps.sh \n", "# Install dependencies from apt\n", "! sudo apt-get install -y libfftw3-dev libsndfile1-dev libgoogle-glog-dev libopenmpi-dev libboost-all-dev\n", "# Install Kenlm\n", "! cd /tmp && git clone https://github.com/kpu/kenlm && cd kenlm && mkdir build && cd build && cmake .. -DCMAKE_BUILD_TYPE=Release && make install -j$(nproc)\n", "\n", "# Install Intel MKL 2020\n", "! cd /tmp && wget https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS-2019.PUB && \\\n", " apt-key add GPG-PUB-KEY-INTEL-SW-PRODUCTS-2019.PUB\n", "! sh -c 'echo deb https://apt.repos.intel.com/mkl all main > /etc/apt/sources.list.d/intel-mkl.list' && \\\n", " apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends intel-mkl-64bit-2020.0-088\n", "# Remove existing MKL libs to avoid double linkeage\n", "! rm -rf /usr/local/lib/libmkl*\n" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "Kao8FS-GjR94", "outputId": "e9d0da1e-266d-4b46-aba4-32f25350fe2b" }, "execution_count": 37, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Reading package lists... Done\n", "Building dependency tree \n", "Reading state information... Done\n", "libboost-all-dev is already the newest version (1.71.0.0ubuntu2).\n", "libopenmpi-dev is already the newest version (4.0.3-0ubuntu1).\n", "libsndfile1-dev is already the newest version (1.0.28-7ubuntu0.1).\n", "The following additional packages will be installed:\n", " libfftw3-bin libfftw3-long3 libfftw3-quad3 libfftw3-single3 libgflags-dev\n", " libgflags2.2 libgoogle-glog0v5\n", "Suggested packages:\n", " libfftw3-doc\n", "The following NEW packages will be installed:\n", " libfftw3-bin libfftw3-dev libfftw3-long3 libfftw3-quad3 libfftw3-single3\n", " libgflags-dev libgflags2.2 libgoogle-glog-dev libgoogle-glog0v5\n", "0 upgraded, 9 newly installed, 0 to remove and 35 not upgraded.\n", "Need to get 4,289 kB of archives.\n", "After this operation, 24.0 MB of additional disk space will be used.\n", "Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libfftw3-long3 amd64 3.3.8-2ubuntu1 [313 kB]\n", "Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libfftw3-quad3 amd64 3.3.8-2ubuntu1 [673 kB]\n", "Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libfftw3-single3 amd64 3.3.8-2ubuntu1 [756 kB]\n", "Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libfftw3-bin amd64 3.3.8-2ubuntu1 [32.2 kB]\n", "Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libfftw3-dev amd64 3.3.8-2ubuntu1 [2,211 kB]\n", "Get:6 http://archive.ubuntu.com/ubuntu focal/universe amd64 libgflags2.2 amd64 2.2.2-1build1 [78.0 kB]\n", "Get:7 http://archive.ubuntu.com/ubuntu focal/universe amd64 libgflags-dev amd64 2.2.2-1build1 [96.6 kB]\n", "Get:8 http://archive.ubuntu.com/ubuntu focal/universe amd64 libgoogle-glog0v5 amd64 0.4.0-1build1 [51.5 kB]\n", "Get:9 http://archive.ubuntu.com/ubuntu focal/universe amd64 libgoogle-glog-dev amd64 0.4.0-1build1 [76.4 kB]\n", "Fetched 4,289 kB in 1s (3,516 kB/s)\n", "debconf: unable to initialize frontend: Dialog\n", "debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 76, <> line 9.)\n", "debconf: falling back to frontend: Readline\n", "debconf: unable to initialize frontend: Readline\n", "debconf: (This frontend requires a controlling tty.)\n", "debconf: falling back to frontend: Teletype\n", "dpkg-preconfigure: unable to re-open stdin: \n", "Selecting previously unselected package libfftw3-long3:amd64.\n", "(Reading database ... 124889 files and directories currently installed.)\n", "Preparing to unpack .../0-libfftw3-long3_3.3.8-2ubuntu1_amd64.deb ...\n", "Unpacking libfftw3-long3:amd64 (3.3.8-2ubuntu1) ...\n", "Selecting previously unselected package libfftw3-quad3:amd64.\n", "Preparing to unpack .../1-libfftw3-quad3_3.3.8-2ubuntu1_amd64.deb ...\n", "Unpacking libfftw3-quad3:amd64 (3.3.8-2ubuntu1) ...\n", "Selecting previously unselected package libfftw3-single3:amd64.\n", "Preparing to unpack .../2-libfftw3-single3_3.3.8-2ubuntu1_amd64.deb ...\n", "Unpacking libfftw3-single3:amd64 (3.3.8-2ubuntu1) ...\n", "Selecting previously unselected package libfftw3-bin.\n", "Preparing to unpack .../3-libfftw3-bin_3.3.8-2ubuntu1_amd64.deb ...\n", "Unpacking libfftw3-bin (3.3.8-2ubuntu1) ...\n", "Selecting previously unselected package libfftw3-dev:amd64.\n", "Preparing to unpack .../4-libfftw3-dev_3.3.8-2ubuntu1_amd64.deb ...\n", "Unpacking libfftw3-dev:amd64 (3.3.8-2ubuntu1) ...\n", "Selecting previously unselected package libgflags2.2.\n", "Preparing to unpack .../5-libgflags2.2_2.2.2-1build1_amd64.deb ...\n", "Unpacking libgflags2.2 (2.2.2-1build1) ...\n", "Selecting previously unselected package libgflags-dev.\n", "Preparing to unpack .../6-libgflags-dev_2.2.2-1build1_amd64.deb ...\n", "Unpacking libgflags-dev (2.2.2-1build1) ...\n", "Selecting previously unselected package libgoogle-glog0v5.\n", "Preparing to unpack .../7-libgoogle-glog0v5_0.4.0-1build1_amd64.deb ...\n", "Unpacking libgoogle-glog0v5 (0.4.0-1build1) ...\n", "Selecting previously unselected package libgoogle-glog-dev.\n", "Preparing to unpack .../8-libgoogle-glog-dev_0.4.0-1build1_amd64.deb ...\n", "Unpacking libgoogle-glog-dev (0.4.0-1build1) ...\n", "Setting up libfftw3-single3:amd64 (3.3.8-2ubuntu1) ...\n", "Setting up libfftw3-long3:amd64 (3.3.8-2ubuntu1) ...\n", "Setting up libfftw3-quad3:amd64 (3.3.8-2ubuntu1) ...\n", "Setting up libgflags2.2 (2.2.2-1build1) ...\n", "Setting up libfftw3-bin (3.3.8-2ubuntu1) ...\n", "Setting up libgflags-dev (2.2.2-1build1) ...\n", "Setting up libfftw3-dev:amd64 (3.3.8-2ubuntu1) ...\n", "Setting up libgoogle-glog0v5 (0.4.0-1build1) ...\n", "Setting up libgoogle-glog-dev (0.4.0-1build1) ...\n", "Processing triggers for man-db (2.9.1-1) ...\n", "Processing triggers for libc-bin (2.31-0ubuntu9.9) ...\n", "Cloning into 'kenlm'...\n", "remote: Enumerating objects: 14147, done.\u001b[K\n", "remote: Counting objects: 100% (460/460), done.\u001b[K\n", "remote: Compressing objects: 100% (319/319), done.\u001b[K\n", "remote: Total 14147 (delta 152), reused 399 (delta 127), pack-reused 13687\u001b[K\n", "Receiving objects: 100% (14147/14147), 5.91 MiB | 15.20 MiB/s, done.\n", "Resolving deltas: 100% (8032/8032), done.\n", "-- The C compiler identification is GNU 9.4.0\n", "-- The CXX compiler identification is GNU 9.4.0\n", "-- Detecting C compiler ABI info\n", "-- Detecting C compiler ABI info - done\n", "-- Check for working C compiler: /usr/bin/cc - skipped\n", "-- Detecting C compile features\n", "-- Detecting C compile features - done\n", "-- Detecting CXX compiler ABI info\n", "-- Detecting CXX compiler ABI info - done\n", "-- Check for working CXX compiler: /usr/bin/c++ - skipped\n", "-- Detecting CXX compile features\n", "-- Detecting CXX compile features - done\n", "-- Could NOT find Eigen3 (missing: Eigen3_DIR)\n", "-- Found Boost: /usr/lib/x86_64-linux-gnu/cmake/Boost-1.71.0/BoostConfig.cmake (found suitable version \"1.71.0\", minimum required is \"1.41.0\") found components: program_options system thread unit_test_framework \n", "-- Check if compiler accepts -pthread\n", "-- Check if compiler accepts -pthread - yes\n", "-- Found Threads: TRUE \n", "-- Found ZLIB: /usr/lib/x86_64-linux-gnu/libz.so (found version \"1.2.11\") \n", "-- Found BZip2: /usr/lib/x86_64-linux-gnu/libbz2.so (found version \"1.0.8\") \n", "-- Looking for BZ2_bzCompressInit\n", "-- Looking for BZ2_bzCompressInit - found\n", "-- Looking for lzma_auto_decoder in /usr/lib/x86_64-linux-gnu/liblzma.so\n", "-- Looking for lzma_auto_decoder in /usr/lib/x86_64-linux-gnu/liblzma.so - found\n", "-- Looking for lzma_easy_encoder in /usr/lib/x86_64-linux-gnu/liblzma.so\n", "-- Looking for lzma_easy_encoder in /usr/lib/x86_64-linux-gnu/liblzma.so - found\n", "-- Looking for lzma_lzma_preset in /usr/lib/x86_64-linux-gnu/liblzma.so\n", "-- Looking for lzma_lzma_preset in /usr/lib/x86_64-linux-gnu/liblzma.so - found\n", "-- Found LibLZMA: /usr/lib/x86_64-linux-gnu/liblzma.so (found version \"5.2.4\") \n", "-- Looking for clock_gettime in rt\n", "-- Looking for clock_gettime in rt - found\n", "-- Configuring done\n", "-- Generating done\n", "-- Build files have been written to: /tmp/kenlm/build\n", "[ 1%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/double-conversion/bignum-dtoa.cc.o\u001b[0m\n", "[ 2%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/double-conversion/bignum.cc.o\u001b[0m\n", "[ 3%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/double-conversion/cached-powers.cc.o\u001b[0m\n", "[ 5%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/double-conversion/fast-dtoa.cc.o\u001b[0m\n", "[ 6%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/double-conversion/fixed-dtoa.cc.o\u001b[0m\n", "[ 7%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/double-conversion/strtod.cc.o\u001b[0m\n", "[ 8%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/double-conversion/double-to-string.cc.o\u001b[0m\n", "[ 10%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/double-conversion/string-to-double.cc.o\u001b[0m\n", "[ 11%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/stream/chain.cc.o\u001b[0m\n", "[ 12%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/stream/count_records.cc.o\u001b[0m\n", "[ 13%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/stream/io.cc.o\u001b[0m\n", "[ 15%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/stream/line_input.cc.o\u001b[0m\n", "[ 16%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/stream/multi_progress.cc.o\u001b[0m\n", "[ 17%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/stream/rewindable_stream.cc.o\u001b[0m\n", "[ 18%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/bit_packing.cc.o\u001b[0m\n", "[ 20%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/ersatz_progress.cc.o\u001b[0m\n", "[ 21%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/exception.cc.o\u001b[0m\n", "[ 22%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/file.cc.o\u001b[0m\n", "[ 23%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/file_piece.cc.o\u001b[0m\n", "[ 25%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/float_to_string.cc.o\u001b[0m\n", "[ 26%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/integer_to_string.cc.o\u001b[0m\n", "[ 27%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/mmap.cc.o\u001b[0m\n", "[ 28%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/murmur_hash.cc.o\u001b[0m\n", "[ 30%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/parallel_read.cc.o\u001b[0m\n", "[ 31%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/pool.cc.o\u001b[0m\n", "[ 32%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/read_compressed.cc.o\u001b[0m\n", "[ 33%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/scoped.cc.o\u001b[0m\n", "[ 35%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/spaces.cc.o\u001b[0m\n", "[ 36%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/string_piece.cc.o\u001b[0m\n", "[ 37%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/usage.cc.o\u001b[0m\n", "[ 38%] \u001b[32m\u001b[1mLinking CXX static library ../lib/libkenlm_util.a\u001b[0m\n", "[ 38%] Built target kenlm_util\n", "[ 40%] \u001b[32mBuilding CXX object util/CMakeFiles/probing_hash_table_benchmark.dir/probing_hash_table_benchmark_main.cc.o\u001b[0m\n", "[ 41%] \u001b[32mBuilding CXX object lm/CMakeFiles/kenlm.dir/bhiksha.cc.o\u001b[0m\n", "[ 42%] \u001b[32mBuilding CXX object lm/CMakeFiles/kenlm.dir/binary_format.cc.o\u001b[0m\n", "[ 43%] \u001b[32mBuilding CXX object lm/CMakeFiles/kenlm.dir/config.cc.o\u001b[0m\n", "[ 45%] \u001b[32mBuilding CXX object lm/CMakeFiles/kenlm.dir/lm_exception.cc.o\u001b[0m\n", "[ 46%] \u001b[32mBuilding CXX object lm/CMakeFiles/kenlm.dir/model.cc.o\u001b[0m\n", "[ 47%] \u001b[32mBuilding CXX object lm/CMakeFiles/kenlm.dir/quantize.cc.o\u001b[0m\n", "[ 48%] \u001b[32mBuilding CXX object lm/CMakeFiles/kenlm.dir/read_arpa.cc.o\u001b[0m\n", "[ 50%] \u001b[32mBuilding CXX object lm/CMakeFiles/kenlm.dir/search_hashed.cc.o\u001b[0m\n", "[ 51%] \u001b[32m\u001b[1mLinking CXX executable ../bin/probing_hash_table_benchmark\u001b[0m\n", "[ 51%] Built target probing_hash_table_benchmark\n", "[ 52%] \u001b[32mBuilding CXX object lm/filter/CMakeFiles/kenlm_filter.dir/arpa_io.cc.o\u001b[0m\n", "[ 53%] \u001b[32mBuilding CXX object lm/filter/CMakeFiles/kenlm_filter.dir/phrase.cc.o\u001b[0m\n", "[ 55%] \u001b[32mBuilding CXX object lm/CMakeFiles/kenlm.dir/search_trie.cc.o\u001b[0m\n", "[ 56%] \u001b[32mBuilding CXX object lm/filter/CMakeFiles/kenlm_filter.dir/vocab.cc.o\u001b[0m\n", "[ 57%] \u001b[32m\u001b[1mLinking CXX static library ../../lib/libkenlm_filter.a\u001b[0m\n", "[ 57%] Built target kenlm_filter\n", "[ 58%] \u001b[32mBuilding CXX object lm/CMakeFiles/kenlm.dir/sizes.cc.o\u001b[0m\n", "[ 60%] \u001b[32mBuilding CXX object lm/CMakeFiles/kenlm.dir/trie.cc.o\u001b[0m\n", "[ 61%] \u001b[32mBuilding CXX object lm/CMakeFiles/kenlm.dir/trie_sort.cc.o\u001b[0m\n", "[ 62%] \u001b[32mBuilding CXX object lm/CMakeFiles/kenlm.dir/value_build.cc.o\u001b[0m\n", "[ 63%] \u001b[32mBuilding CXX object lm/CMakeFiles/kenlm.dir/virtual_interface.cc.o\u001b[0m\n", "[ 65%] \u001b[32mBuilding CXX object lm/CMakeFiles/kenlm.dir/vocab.cc.o\u001b[0m\n", "[ 66%] \u001b[32mBuilding CXX object lm/CMakeFiles/kenlm.dir/common/model_buffer.cc.o\u001b[0m\n", "[ 67%] \u001b[32mBuilding CXX object lm/CMakeFiles/kenlm.dir/common/print.cc.o\u001b[0m\n", "[ 68%] \u001b[32mBuilding CXX object lm/CMakeFiles/kenlm.dir/common/renumber.cc.o\u001b[0m\n", "[ 70%] \u001b[32mBuilding CXX object lm/CMakeFiles/kenlm.dir/common/size_option.cc.o\u001b[0m\n", "[ 71%] \u001b[32m\u001b[1mLinking CXX static library ../lib/libkenlm.a\u001b[0m\n", "[ 71%] Built target kenlm\n", "[ 72%] \u001b[32mBuilding CXX object lm/CMakeFiles/query.dir/query_main.cc.o\u001b[0m\n", "[ 73%] \u001b[32mBuilding CXX object lm/CMakeFiles/fragment.dir/fragment_main.cc.o\u001b[0m\n", "[ 75%] \u001b[32m\u001b[1mLinking CXX executable ../bin/fragment\u001b[0m\n", "[ 75%] Built target fragment\n", "[ 76%] \u001b[32mBuilding CXX object lm/CMakeFiles/build_binary.dir/build_binary_main.cc.o\u001b[0m\n", "[ 77%] \u001b[32m\u001b[1mLinking CXX executable ../bin/query\u001b[0m\n", "[ 77%] Built target query\n", "[ 78%] \u001b[32mBuilding CXX object lm/CMakeFiles/kenlm_benchmark.dir/kenlm_benchmark_main.cc.o\u001b[0m\n", "[ 80%] \u001b[32m\u001b[1mLinking CXX executable ../bin/build_binary\u001b[0m\n", "[ 80%] Built target build_binary\n", "[ 81%] \u001b[32mBuilding CXX object lm/builder/CMakeFiles/kenlm_builder.dir/adjust_counts.cc.o\u001b[0m\n", "[ 82%] \u001b[32mBuilding CXX object lm/builder/CMakeFiles/kenlm_builder.dir/corpus_count.cc.o\u001b[0m\n", "[ 83%] \u001b[32mBuilding CXX object lm/builder/CMakeFiles/kenlm_builder.dir/initial_probabilities.cc.o\u001b[0m\n", "[ 85%] \u001b[32mBuilding CXX object lm/builder/CMakeFiles/kenlm_builder.dir/interpolate.cc.o\u001b[0m\n", "[ 86%] \u001b[32mBuilding CXX object lm/builder/CMakeFiles/kenlm_builder.dir/output.cc.o\u001b[0m\n", "[ 87%] \u001b[32m\u001b[1mLinking CXX executable ../bin/kenlm_benchmark\u001b[0m\n", "[ 87%] Built target kenlm_benchmark\n", "[ 88%] \u001b[32mBuilding CXX object lm/filter/CMakeFiles/filter.dir/filter_main.cc.o\u001b[0m\n", "[ 90%] \u001b[32mBuilding CXX object lm/builder/CMakeFiles/kenlm_builder.dir/pipeline.cc.o\u001b[0m\n", "[ 91%] \u001b[32m\u001b[1mLinking CXX static library ../../lib/libkenlm_builder.a\u001b[0m\n", "[ 91%] Built target kenlm_builder\n", "[ 92%] \u001b[32mBuilding CXX object lm/filter/CMakeFiles/phrase_table_vocab.dir/phrase_table_vocab_main.cc.o\u001b[0m\n", "[ 93%] \u001b[32m\u001b[1mLinking CXX executable ../../bin/phrase_table_vocab\u001b[0m\n", "[ 93%] Built target phrase_table_vocab\n", "[ 95%] \u001b[32mBuilding CXX object lm/builder/CMakeFiles/lmplz.dir/lmplz_main.cc.o\u001b[0m\n", "[ 96%] \u001b[32m\u001b[1mLinking CXX executable ../../bin/filter\u001b[0m\n", "[ 96%] Built target filter\n", "[ 97%] \u001b[32mBuilding CXX object lm/builder/CMakeFiles/count_ngrams.dir/count_ngrams_main.cc.o\u001b[0m\n", "[ 98%] \u001b[32m\u001b[1mLinking CXX executable ../../bin/lmplz\u001b[0m\n", "[ 98%] Built target lmplz\n", "[100%] \u001b[32m\u001b[1mLinking CXX executable ../../bin/count_ngrams\u001b[0m\n", "[100%] Built target count_ngrams\n", "\u001b[36mInstall the project...\u001b[0m\n", "-- Install configuration: \"Release\"\n", "-- Installing: /usr/local/share/kenlm/cmake/kenlmTargets.cmake\n", "-- Installing: /usr/local/share/kenlm/cmake/kenlmTargets-release.cmake\n", "-- Installing: /usr/local/include/kenlm/util/bit_packing.hh\n", "-- Installing: /usr/local/include/kenlm/util/ersatz_progress.hh\n", "-- Installing: /usr/local/include/kenlm/util/exception.hh\n", "-- Installing: /usr/local/include/kenlm/util/fake_ostream.hh\n", "-- Installing: /usr/local/include/kenlm/util/file.hh\n", "-- Installing: /usr/local/include/kenlm/util/file_piece.hh\n", "-- Installing: /usr/local/include/kenlm/util/file_stream.hh\n", "-- Installing: /usr/local/include/kenlm/util/fixed_array.hh\n", "-- Installing: /usr/local/include/kenlm/util/float_to_string.hh\n", "-- Installing: /usr/local/include/kenlm/util/getopt.hh\n", "-- Installing: /usr/local/include/kenlm/util/have.hh\n", "-- Installing: /usr/local/include/kenlm/util/integer_to_string.hh\n", "-- Installing: /usr/local/include/kenlm/util/joint_sort.hh\n", "-- Installing: /usr/local/include/kenlm/util/mmap.hh\n", "-- Installing: /usr/local/include/kenlm/util/multi_intersection.hh\n", "-- Installing: /usr/local/include/kenlm/util/murmur_hash.hh\n", "-- Installing: /usr/local/include/kenlm/util/parallel_read.hh\n", "-- Installing: /usr/local/include/kenlm/util/pcqueue.hh\n", "-- Installing: /usr/local/include/kenlm/util/pool.hh\n", "-- Installing: /usr/local/include/kenlm/util/probing_hash_table.hh\n", "-- Installing: /usr/local/include/kenlm/util/proxy_iterator.hh\n", "-- Installing: /usr/local/include/kenlm/util/read_compressed.hh\n", "-- Installing: /usr/local/include/kenlm/util/scoped.hh\n", "-- Installing: /usr/local/include/kenlm/util/sized_iterator.hh\n", "-- Installing: /usr/local/include/kenlm/util/sorted_uniform.hh\n", "-- Installing: /usr/local/include/kenlm/util/spaces.hh\n", "-- Installing: /usr/local/include/kenlm/util/string_piece.hh\n", "-- Installing: /usr/local/include/kenlm/util/string_piece_hash.hh\n", "-- Installing: /usr/local/include/kenlm/util/string_stream.hh\n", "-- Installing: /usr/local/include/kenlm/util/thread_pool.hh\n", "-- Installing: /usr/local/include/kenlm/util/tokenize_piece.hh\n", "-- Installing: /usr/local/include/kenlm/util/usage.hh\n", "-- Installing: /usr/local/include/kenlm/util/double-conversion/bignum-dtoa.h\n", "-- Installing: /usr/local/include/kenlm/util/double-conversion/bignum.h\n", "-- Installing: /usr/local/include/kenlm/util/double-conversion/cached-powers.h\n", "-- Installing: /usr/local/include/kenlm/util/double-conversion/diy-fp.h\n", "-- Installing: /usr/local/include/kenlm/util/double-conversion/double-conversion.h\n", "-- Installing: /usr/local/include/kenlm/util/double-conversion/double-to-string.h\n", "-- Installing: /usr/local/include/kenlm/util/double-conversion/fast-dtoa.h\n", "-- Installing: /usr/local/include/kenlm/util/double-conversion/fixed-dtoa.h\n", "-- Installing: /usr/local/include/kenlm/util/double-conversion/ieee.h\n", "-- Installing: /usr/local/include/kenlm/util/double-conversion/string-to-double.h\n", "-- Installing: /usr/local/include/kenlm/util/double-conversion/strtod.h\n", "-- Installing: /usr/local/include/kenlm/util/double-conversion/utils.h\n", "-- Installing: /usr/local/include/kenlm/util/stream/block.hh\n", "-- Installing: /usr/local/include/kenlm/util/stream/chain.hh\n", "-- Installing: /usr/local/include/kenlm/util/stream/config.hh\n", "-- Installing: /usr/local/include/kenlm/util/stream/count_records.hh\n", "-- Installing: /usr/local/include/kenlm/util/stream/io.hh\n", "-- Installing: /usr/local/include/kenlm/util/stream/line_input.hh\n", "-- Installing: /usr/local/include/kenlm/util/stream/multi_progress.hh\n", "-- Installing: /usr/local/include/kenlm/util/stream/multi_stream.hh\n", "-- Installing: /usr/local/include/kenlm/util/stream/rewindable_stream.hh\n", "-- Installing: /usr/local/include/kenlm/util/stream/sort.hh\n", "-- Installing: /usr/local/include/kenlm/util/stream/stream.hh\n", "-- Installing: /usr/local/include/kenlm/util/stream/typed_stream.hh\n", "-- Installing: /usr/local/include/kenlm/lm/bhiksha.hh\n", "-- Installing: /usr/local/include/kenlm/lm/binary_format.hh\n", "-- Installing: /usr/local/include/kenlm/lm/blank.hh\n", "-- Installing: /usr/local/include/kenlm/lm/config.hh\n", "-- Installing: /usr/local/include/kenlm/lm/enumerate_vocab.hh\n", "-- Installing: /usr/local/include/kenlm/lm/facade.hh\n", "-- Installing: /usr/local/include/kenlm/lm/left.hh\n", "-- Installing: /usr/local/include/kenlm/lm/lm_exception.hh\n", "-- Installing: /usr/local/include/kenlm/lm/max_order.hh\n", "-- Installing: /usr/local/include/kenlm/lm/model.hh\n", "-- Installing: /usr/local/include/kenlm/lm/model_type.hh\n", "-- Installing: /usr/local/include/kenlm/lm/ngram_query.hh\n", "-- Installing: /usr/local/include/kenlm/lm/partial.hh\n", "-- Installing: /usr/local/include/kenlm/lm/quantize.hh\n", "-- Installing: /usr/local/include/kenlm/lm/read_arpa.hh\n", "-- Installing: /usr/local/include/kenlm/lm/return.hh\n", "-- Installing: /usr/local/include/kenlm/lm/search_hashed.hh\n", "-- Installing: /usr/local/include/kenlm/lm/search_trie.hh\n", "-- Installing: /usr/local/include/kenlm/lm/sizes.hh\n", "-- Installing: /usr/local/include/kenlm/lm/state.hh\n", "-- Installing: /usr/local/include/kenlm/lm/trie.hh\n", "-- Installing: /usr/local/include/kenlm/lm/trie_sort.hh\n", "-- Installing: /usr/local/include/kenlm/lm/value.hh\n", "-- Installing: /usr/local/include/kenlm/lm/value_build.hh\n", "-- Installing: /usr/local/include/kenlm/lm/virtual_interface.hh\n", "-- Installing: /usr/local/include/kenlm/lm/vocab.hh\n", "-- Installing: /usr/local/include/kenlm/lm/weights.hh\n", "-- Installing: /usr/local/include/kenlm/lm/word_index.hh\n", "-- Installing: /usr/local/include/kenlm/lm/builder/adjust_counts.hh\n", "-- Installing: /usr/local/include/kenlm/lm/builder/combine_counts.hh\n", "-- Installing: /usr/local/include/kenlm/lm/builder/corpus_count.hh\n", "-- Installing: /usr/local/include/kenlm/lm/builder/debug_print.hh\n", "-- Installing: /usr/local/include/kenlm/lm/builder/discount.hh\n", "-- Installing: /usr/local/include/kenlm/lm/builder/hash_gamma.hh\n", "-- Installing: /usr/local/include/kenlm/lm/builder/header_info.hh\n", "-- Installing: /usr/local/include/kenlm/lm/builder/initial_probabilities.hh\n", "-- Installing: /usr/local/include/kenlm/lm/builder/interpolate.hh\n", "-- Installing: /usr/local/include/kenlm/lm/builder/output.hh\n", "-- Installing: /usr/local/include/kenlm/lm/builder/payload.hh\n", "-- Installing: /usr/local/include/kenlm/lm/builder/pipeline.hh\n", "-- Installing: /usr/local/include/kenlm/lm/common/compare.hh\n", "-- Installing: /usr/local/include/kenlm/lm/common/joint_order.hh\n", "-- Installing: /usr/local/include/kenlm/lm/common/model_buffer.hh\n", "-- Installing: /usr/local/include/kenlm/lm/common/ngram.hh\n", "-- Installing: /usr/local/include/kenlm/lm/common/ngram_stream.hh\n", "-- Installing: /usr/local/include/kenlm/lm/common/print.hh\n", "-- Installing: /usr/local/include/kenlm/lm/common/renumber.hh\n", "-- Installing: /usr/local/include/kenlm/lm/common/size_option.hh\n", "-- Installing: /usr/local/include/kenlm/lm/common/special.hh\n", "-- Installing: /usr/local/include/kenlm/lm/filter/arpa_io.hh\n", "-- Installing: /usr/local/include/kenlm/lm/filter/count_io.hh\n", "-- Installing: /usr/local/include/kenlm/lm/filter/format.hh\n", "-- Installing: /usr/local/include/kenlm/lm/filter/phrase.hh\n", "-- Installing: /usr/local/include/kenlm/lm/filter/thread.hh\n", "-- Installing: /usr/local/include/kenlm/lm/filter/vocab.hh\n", "-- Installing: /usr/local/include/kenlm/lm/filter/wrapper.hh\n", "-- Installing: /usr/local/include/kenlm/lm/interpolate/backoff_matrix.hh\n", "-- Installing: /usr/local/include/kenlm/lm/interpolate/backoff_reunification.hh\n", "-- Installing: /usr/local/include/kenlm/lm/interpolate/bounded_sequence_encoding.hh\n", "-- Installing: /usr/local/include/kenlm/lm/interpolate/interpolate_info.hh\n", "-- Installing: /usr/local/include/kenlm/lm/interpolate/merge_probabilities.hh\n", "-- Installing: /usr/local/include/kenlm/lm/interpolate/merge_vocab.hh\n", "-- Installing: /usr/local/include/kenlm/lm/interpolate/normalize.hh\n", "-- Installing: /usr/local/include/kenlm/lm/interpolate/pipeline.hh\n", "-- Installing: /usr/local/include/kenlm/lm/interpolate/split_worker.hh\n", "-- Installing: /usr/local/include/kenlm/lm/interpolate/tune_derivatives.hh\n", "-- Installing: /usr/local/include/kenlm/lm/interpolate/tune_instances.hh\n", "-- Installing: /usr/local/include/kenlm/lm/interpolate/tune_matrix.hh\n", "-- Installing: /usr/local/include/kenlm/lm/interpolate/tune_weights.hh\n", "-- Installing: /usr/local/include/kenlm/lm/interpolate/universal_vocab.hh\n", "-- Installing: /usr/local/share/kenlm/cmake/kenlmConfig.cmake\n", "-- Installing: /usr/local/lib/libkenlm_util.a\n", "-- Installing: /usr/local/bin/probing_hash_table_benchmark\n", "-- Installing: /usr/local/lib/libkenlm.a\n", "-- Installing: /usr/local/bin/query\n", "-- Installing: /usr/local/bin/fragment\n", "-- Installing: /usr/local/bin/build_binary\n", "-- Installing: /usr/local/bin/kenlm_benchmark\n", "-- Installing: /usr/local/bin/lmplz\n", "-- Installing: /usr/local/bin/count_ngrams\n", "-- Installing: /usr/local/lib/libkenlm_builder.a\n", "-- Installing: /usr/local/bin/filter\n", "-- Installing: /usr/local/bin/phrase_table_vocab\n", "-- Installing: /usr/local/lib/libkenlm_filter.a\n" ] } ] }, { "cell_type": "code", "source": [ "! rm -rf flashlight\n", "! git clone --recursive https://github.com/flashlight/flashlight.git\n", "%cd flashlight\n", "! git checkout 035ead6efefb82b47c8c2e643603e87d38850076 \n", "%cd bindings/python \n", "! python3 setup.py install\n", "\n", "%cd /content/fairseq " ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "5LjIgBfzhQ4w", "outputId": "5d1023c5-f6ad-46b1-9326-b51965c722c3" }, "execution_count": 38, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Cloning into 'flashlight'...\n", "remote: Enumerating objects: 24032, done.\u001b[K\n", "remote: Counting objects: 100% (150/150), done.\u001b[K\n", "remote: Compressing objects: 100% (123/123), done.\u001b[K\n", "remote: Total 24032 (delta 41), reused 111 (delta 24), pack-reused 23882\u001b[K\n", "Receiving objects: 100% (24032/24032), 15.30 MiB | 2.64 MiB/s, done.\n", "Resolving deltas: 100% (17089/17089), done.\n", "/content/fairseq/flashlight\n", "Note: switching to '035ead6efefb82b47c8c2e643603e87d38850076'.\n", "\n", "You are in 'detached HEAD' state. You can look around, make experimental\n", "changes and commit them, and you can discard any commits you make in this\n", "state without impacting any branches by switching back to a branch.\n", "\n", "If you want to create a new branch to retain commits you create, you may\n", "do so (now or later) by using -c with the switch command. Example:\n", "\n", " git switch -c \n", "\n", "Or undo this operation with:\n", "\n", " git switch -\n", "\n", "Turn off this advice by setting config variable advice.detachedHead to false\n", "\n", "HEAD is now at 035ead6e AdvancedIndex fix\n", "/content/fairseq/flashlight/bindings/python\n", "running install\n", "/usr/local/lib/python3.10/dist-packages/setuptools/_distutils/cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.\n", "!!\n", "\n", " ********************************************************************************\n", " Please avoid running ``setup.py`` directly.\n", " Instead, use pypa/build, pypa/installer, pypa/build or\n", " other standards-based tools.\n", "\n", " See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.\n", " ********************************************************************************\n", "\n", "!!\n", " self.initialize_options()\n", "/usr/local/lib/python3.10/dist-packages/setuptools/_distutils/cmd.py:66: EasyInstallDeprecationWarning: easy_install command is deprecated.\n", "!!\n", "\n", " ********************************************************************************\n", " Please avoid running ``setup.py`` and ``easy_install``.\n", " Instead, use pypa/build, pypa/installer, pypa/build or\n", " other standards-based tools.\n", "\n", " See https://github.com/pypa/setuptools/issues/917 for details.\n", " ********************************************************************************\n", "\n", "!!\n", " self.initialize_options()\n", "running bdist_egg\n", "running egg_info\n", "creating flashlight.egg-info\n", "writing flashlight.egg-info/PKG-INFO\n", "writing dependency_links to flashlight.egg-info/dependency_links.txt\n", "writing top-level names to flashlight.egg-info/top_level.txt\n", "writing manifest file 'flashlight.egg-info/SOURCES.txt'\n", "reading manifest file 'flashlight.egg-info/SOURCES.txt'\n", "reading manifest template 'MANIFEST.in'\n", "writing manifest file 'flashlight.egg-info/SOURCES.txt'\n", "installing library code to build/bdist.linux-x86_64/egg\n", "running install_lib\n", "running build_py\n", "creating build\n", "creating build/lib.linux-x86_64-cpython-310\n", "creating build/lib.linux-x86_64-cpython-310/flashlight\n", "creating build/lib.linux-x86_64-cpython-310/flashlight/lib\n", "copying flashlight/lib/__init__.py -> build/lib.linux-x86_64-cpython-310/flashlight/lib\n", "creating build/lib.linux-x86_64-cpython-310/flashlight/lib/audio\n", "copying flashlight/lib/audio/feature.py -> build/lib.linux-x86_64-cpython-310/flashlight/lib/audio\n", "creating build/lib.linux-x86_64-cpython-310/flashlight/lib/sequence\n", "copying flashlight/lib/sequence/criterion_torch.py -> build/lib.linux-x86_64-cpython-310/flashlight/lib/sequence\n", "copying flashlight/lib/sequence/criterion.py -> build/lib.linux-x86_64-cpython-310/flashlight/lib/sequence\n", "creating build/lib.linux-x86_64-cpython-310/flashlight/lib/text\n", "copying flashlight/lib/text/dictionary.py -> build/lib.linux-x86_64-cpython-310/flashlight/lib/text\n", "copying flashlight/lib/text/decoder.py -> build/lib.linux-x86_64-cpython-310/flashlight/lib/text\n", "running build_ext\n", "-- The CXX compiler identification is GNU 9.4.0\n", "-- The C compiler identification is GNU 9.4.0\n", "-- Detecting CXX compiler ABI info\n", "-- Detecting CXX compiler ABI info - done\n", "-- Check for working CXX compiler: /usr/bin/c++ - skipped\n", "-- Detecting CXX compile features\n", "-- Detecting CXX compile features - done\n", "-- Detecting C compiler ABI info\n", "-- Detecting C compiler ABI info - done\n", "-- Check for working C compiler: /usr/bin/cc - skipped\n", "-- Detecting C compile features\n", "-- Detecting C compile features - done\n", "-- Performing Test COMPILER_SUPPORTS_RDYNAMIC\n", "-- Performing Test COMPILER_SUPPORTS_RDYNAMIC - Success\n", "-- -rdynamic supported.\n", "-- The CUDA compiler identification is NVIDIA 11.8.89\n", "-- Detecting CUDA compiler ABI info\n", "-- Detecting CUDA compiler ABI info - done\n", "-- Check for working CUDA compiler: /usr/local/cuda/bin/nvcc - skipped\n", "-- Detecting CUDA compile features\n", "-- Detecting CUDA compile features - done\n", "-- Performing Test CMAKE_HAVE_LIBC_PTHREAD\n", "-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed\n", "-- Looking for pthread_create in pthreads\n", "-- Looking for pthread_create in pthreads - not found\n", "-- Looking for pthread_create in pthread\n", "-- Looking for pthread_create in pthread - found\n", "-- Found Threads: TRUE \n", "-- CUDA found (library: /usr/local/cuda/lib64/libcudart_static.a;Threads::Threads;dl;/usr/lib/x86_64-linux-gnu/librt.so include: /usr/local/cuda/include)\n", "-- CUDA architecture flags: -gencodearch=compute_35,code=sm_35-gencodearch=compute_50,code=sm_50-gencodearch=compute_52,code=sm_52-gencodearch=compute_60,code=sm_60-gencodearch=compute_61,code=sm_61-gencodearch=compute_70,code=sm_70-gencodearch=compute_75,code=sm_75-gencodearch=compute_80,code=sm_80-gencodearch=compute_80,code=compute_80\n", "-- Will build flashlight libraries.\n", "-- MKL_THREADING = OMP\n", "-- Looking for sys/types.h\n", "-- Looking for sys/types.h - found\n", "-- Looking for stdint.h\n", "-- Looking for stdint.h - found\n", "-- Looking for stddef.h\n", "-- Looking for stddef.h - found\n", "-- Check size of void*\n", "-- Check size of void* - done\n", "-- Checking for [mkl_intel_lp64 - mkl_gnu_thread - mkl_core - gomp - pthread - m - dl]\n", "-- Library mkl_intel_lp64: /opt/intel/mkl/lib/intel64/libmkl_intel_lp64.so\n", "-- Library mkl_gnu_thread: /opt/intel/mkl/lib/intel64/libmkl_gnu_thread.so\n", "-- Library mkl_core: /opt/intel/mkl/lib/intel64/libmkl_core.so\n", "-- Library gomp: -fopenmp\n", "-- Library pthread: /usr/lib/x86_64-linux-gnu/libpthread.so\n", "-- Library m: /usr/lib/x86_64-linux-gnu/libm.so\n", "-- Library dl: /usr/lib/x86_64-linux-gnu/libdl.so\n", "-- Looking for cblas_sgemm\n", "-- Looking for cblas_sgemm - found\n", "-- MKL library found\n", "-- CBLAS found (include: /opt/intel/mkl/include, library: /opt/intel/mkl/lib/intel64/libmkl_intel_lp64.so;/opt/intel/mkl/lib/intel64/libmkl_gnu_thread.so;/opt/intel/mkl/lib/intel64/libmkl_core.so;-fopenmp;/usr/lib/x86_64-linux-gnu/libpthread.so;/usr/lib/x86_64-linux-gnu/libm.so;/usr/lib/x86_64-linux-gnu/libdl.so)\n", "-- Could NOT find FFTW3 (missing: FFTW3_DIR)\n", "-- Found PkgConfig: /usr/bin/pkg-config (found version \"0.29.1\") \n", "-- FindFFTW using pkgconfig: FOUND=1 LIBRARIES=fftw3 LIBRARY_DIRS=/usr/lib/x86_64-linux-gnu LIBDIR=/usr/lib/x86_64-linux-gnu LINK_LIBRARIES=/usr/lib/x86_64-linux-gnu/libfftw3.so\n", "-- FindFTTW using pkgconfig: INCLUDE_DIRS= INCLUDEDIR=/usr/include\n", "-- Found FFTW3: /usr/include \n", "-- FFTW found\n", "-- Found OpenMP_C: -fopenmp (found version \"4.5\") \n", "-- Found OpenMP_CXX: -fopenmp (found version \"4.5\") \n", "-- Found OpenMP: TRUE (found version \"4.5\") \n", "-- Looking for KenLM\n", "-- Looking for lzma_auto_decoder in /usr/lib/x86_64-linux-gnu/liblzma.so\n", "-- Looking for lzma_auto_decoder in /usr/lib/x86_64-linux-gnu/liblzma.so - found\n", "-- Looking for lzma_easy_encoder in /usr/lib/x86_64-linux-gnu/liblzma.so\n", "-- Looking for lzma_easy_encoder in /usr/lib/x86_64-linux-gnu/liblzma.so - found\n", "-- Looking for lzma_lzma_preset in /usr/lib/x86_64-linux-gnu/liblzma.so\n", "-- Looking for lzma_lzma_preset in /usr/lib/x86_64-linux-gnu/liblzma.so - found\n", "-- Found LibLZMA: /usr/lib/x86_64-linux-gnu/liblzma.so (found version \"5.2.4\") \n", "-- Found BZip2: /usr/lib/x86_64-linux-gnu/libbz2.so (found version \"1.0.8\") \n", "-- Looking for BZ2_bzCompressInit\n", "-- Looking for BZ2_bzCompressInit - found\n", "-- Found ZLIB: /usr/lib/x86_64-linux-gnu/libz.so (found version \"1.2.11\") \n", "-- Using kenlm library found in /usr/local/lib/libkenlm.a\n", "-- Using kenlm utils library found in /usr/local/lib/libkenlm_util.a\n", "-- kenlm model.hh found in /usr/local/include/kenlm/lm\n", "-- Found kenlm: /usr/local/include \n", "-- Found kenlm (include: /usr/local/include, library: /usr/local/lib/libkenlm.a;/usr/local/lib/libkenlm_util.a;/usr/lib/x86_64-linux-gnu/liblzma.so;/usr/lib/x86_64-linux-gnu/libbz2.so;/usr/lib/x86_64-linux-gnu/libz.so)\n", "-- Found PythonInterp: /usr/bin/python3 (found version \"3.10.11\") \n", "-- Found PythonLibs: /usr/lib/x86_64-linux-gnu/libpython3.10.so\n", "-- Performing Test HAS_FLTO\n", "-- Performing Test HAS_FLTO - Success\n", "-- LTO enabled\n", "-- Configuring done\n", "\u001b[33mCMake Warning (dev) in CMakeLists.txt:\n", " Policy CMP0104 is not set: CMAKE_CUDA_ARCHITECTURES now detected for NVCC,\n", " empty CUDA_ARCHITECTURES not allowed. Run \"cmake --help-policy CMP0104\"\n", " for policy details. Use the cmake_policy command to set the policy and\n", " suppress this warning.\n", "\n", " CUDA_ARCHITECTURES is empty for target \"fl-libraries\".\n", "This warning is for project developers. Use -Wno-dev to suppress it.\n", "\u001b[0m\n", "-- Generating done\n", "-- Build files have been written to: /content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310\n", "[ 1%] \u001b[34m\u001b[1mCreating directories for 'pybind11'\u001b[0m\n", "[ 3%] \u001b[32mBuilding CXX object CMakeFiles/fl-libraries.dir/flashlight/lib/set/Hungarian.cpp.o\u001b[0m\n", "[ 5%] \u001b[32mBuilding CXX object CMakeFiles/fl-libraries.dir/flashlight/lib/sequence/criterion/cpu/CriterionUtils.cpp.o\u001b[0m\n", "[ 7%] \u001b[32mBuilding CXX object CMakeFiles/fl-libraries.dir/flashlight/lib/sequence/criterion/cpu/ForceAlignmentCriterion.cpp.o\u001b[0m\n", "[ 9%] \u001b[34m\u001b[1mPerforming download step (git clone) for 'pybind11'\u001b[0m\n", "Cloning into 'pybind11'...\n", "[ 11%] \u001b[32mBuilding CXX object CMakeFiles/fl-libraries.dir/flashlight/lib/sequence/criterion/cpu/ConnectionistTemporalClassificationCriterion.cpp.o\u001b[0m\n", "[ 13%] \u001b[32mBuilding CXX object CMakeFiles/fl-libraries.dir/flashlight/lib/sequence/criterion/cpu/FullConnectionCriterion.cpp.o\u001b[0m\n", "[ 15%] \u001b[32mBuilding CXX object CMakeFiles/fl-libraries.dir/flashlight/lib/sequence/criterion/cpu/ViterbiPath.cpp.o\u001b[0m\n", "[ 16%] \u001b[32mBuilding CXX object CMakeFiles/fl-libraries.dir/flashlight/lib/audio/feature/Ceplifter.cpp.o\u001b[0m\n", "[ 18%] \u001b[32mBuilding CXX object CMakeFiles/fl-libraries.dir/flashlight/lib/audio/feature/Dct.cpp.o\u001b[0m\n", "[ 20%] \u001b[32mBuilding CXX object CMakeFiles/fl-libraries.dir/flashlight/lib/audio/feature/Derivatives.cpp.o\u001b[0m\n", "HEAD is now at 9a19306f bump version to 2.2.4\n", "Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'tools/clang'\n", "Cloning into '/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/tools/clang'...\n", "[ 22%] \u001b[32mBuilding CXX object CMakeFiles/fl-libraries.dir/flashlight/lib/audio/feature/Dither.cpp.o\u001b[0m\n", "[ 24%] \u001b[32mBuilding CXX object CMakeFiles/fl-libraries.dir/flashlight/lib/audio/feature/Mfcc.cpp.o\u001b[0m\n", "[ 26%] \u001b[32mBuilding CXX object CMakeFiles/fl-libraries.dir/flashlight/lib/audio/feature/Mfsc.cpp.o\u001b[0m\n", "Submodule path 'tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'\n", "[ 28%] \u001b[34m\u001b[1mPerforming update step for 'pybind11'\u001b[0m\n", "[ 30%] \u001b[34m\u001b[1mNo patch step for 'pybind11'\u001b[0m\n", "[ 32%] \u001b[34m\u001b[1mNo configure step for 'pybind11'\u001b[0m\n", "[ 33%] \u001b[34m\u001b[1mNo build step for 'pybind11'\u001b[0m\n", "[ 35%] \u001b[34m\u001b[1mNo install step for 'pybind11'\u001b[0m\n", "[ 37%] \u001b[34m\u001b[1mCompleted 'pybind11'\u001b[0m\n", "[ 37%] Built target pybind11\n", "[ 39%] \u001b[32mBuilding CXX object CMakeFiles/fl-libraries.dir/flashlight/lib/audio/feature/PowerSpectrum.cpp.o\u001b[0m\n", "[ 41%] \u001b[32mBuilding CXX object CMakeFiles/fl-libraries.dir/flashlight/lib/audio/feature/PreEmphasis.cpp.o\u001b[0m\n", "[ 43%] \u001b[32mBuilding CXX object CMakeFiles/fl-libraries.dir/flashlight/lib/audio/feature/SpeechUtils.cpp.o\u001b[0m\n", "[ 45%] \u001b[32mBuilding CXX object CMakeFiles/fl-libraries.dir/flashlight/lib/audio/feature/TriFilterbank.cpp.o\u001b[0m\n", "[ 47%] \u001b[32mBuilding CXX object CMakeFiles/fl-libraries.dir/flashlight/lib/audio/feature/Windowing.cpp.o\u001b[0m\n", "[ 49%] \u001b[32mBuilding CXX object CMakeFiles/fl-libraries.dir/flashlight/lib/common/String.cpp.o\u001b[0m\n", "[ 50%] \u001b[32mBuilding CXX object CMakeFiles/fl-libraries.dir/flashlight/lib/common/System.cpp.o\u001b[0m\n", "[ 52%] \u001b[32mBuilding CXX object CMakeFiles/fl-libraries.dir/flashlight/lib/text/decoder/lm/ConvLM.cpp.o\u001b[0m\n", "[ 54%] \u001b[32mBuilding CXX object CMakeFiles/fl-libraries.dir/flashlight/lib/text/decoder/lm/ZeroLM.cpp.o\u001b[0m\n", "[ 56%] \u001b[32mBuilding CXX object CMakeFiles/fl-libraries.dir/flashlight/lib/text/decoder/lm/KenLM.cpp.o\u001b[0m\n", "[ 58%] \u001b[32mBuilding CXX object CMakeFiles/fl-libraries.dir/flashlight/lib/text/decoder/LexiconDecoder.cpp.o\u001b[0m\n", "[ 60%] \u001b[32mBuilding CXX object CMakeFiles/fl-libraries.dir/flashlight/lib/text/decoder/LexiconFreeDecoder.cpp.o\u001b[0m\n", "[ 62%] \u001b[32mBuilding CXX object CMakeFiles/fl-libraries.dir/flashlight/lib/text/decoder/LexiconSeq2SeqDecoder.cpp.o\u001b[0m\n", "[ 64%] \u001b[32mBuilding CXX object CMakeFiles/fl-libraries.dir/flashlight/lib/text/decoder/LexiconFreeSeq2SeqDecoder.cpp.o\u001b[0m\n", "[ 66%] \u001b[32mBuilding CXX object CMakeFiles/fl-libraries.dir/flashlight/lib/text/decoder/Trie.cpp.o\u001b[0m\n", "[ 67%] \u001b[32mBuilding CXX object CMakeFiles/fl-libraries.dir/flashlight/lib/text/decoder/Utils.cpp.o\u001b[0m\n", "[ 69%] \u001b[32mBuilding CXX object CMakeFiles/fl-libraries.dir/flashlight/lib/text/dictionary/Dictionary.cpp.o\u001b[0m\n", "[ 71%] \u001b[32mBuilding CXX object CMakeFiles/fl-libraries.dir/flashlight/lib/text/dictionary/Utils.cpp.o\u001b[0m\n", "[ 73%] \u001b[32mBuilding CXX object CMakeFiles/fl-libraries.dir/flashlight/lib/text/tokenizer/PartialFileReader.cpp.o\u001b[0m\n", "[ 75%] \u001b[32mBuilding CXX object CMakeFiles/fl-libraries.dir/flashlight/lib/text/tokenizer/Tokenizer.cpp.o\u001b[0m\n", "[ 77%] \u001b[32mBuilding CUDA object CMakeFiles/fl-libraries.dir/flashlight/lib/sequence/criterion/cuda/CriterionUtils.cu.o\u001b[0m\n", "[ 79%] \u001b[32mBuilding CUDA object CMakeFiles/fl-libraries.dir/flashlight/lib/sequence/criterion/cuda/ForceAlignmentCriterion.cu.o\u001b[0m\n", "[ 81%] \u001b[32mBuilding CUDA object CMakeFiles/fl-libraries.dir/flashlight/lib/sequence/criterion/cuda/FullConnectionCriterion.cu.o\u001b[0m\n", "[ 83%] \u001b[32mBuilding CUDA object CMakeFiles/fl-libraries.dir/flashlight/lib/sequence/criterion/cuda/ViterbiPath.cu.o\u001b[0m\n", "[ 84%] \u001b[32m\u001b[1mLinking CXX shared library /content/fairseq/flashlight/bindings/python/build/lib.linux-x86_64-cpython-310/libfl-libraries.so\u001b[0m\n", "[ 84%] Built target fl-libraries\n", "[ 86%] \u001b[32mBuilding CXX object CMakeFiles/flashlight_lib_audio_feature.dir/bindings/python/flashlight/lib/audio/_feature.cpp.o\u001b[0m\n", "[ 88%] \u001b[32mBuilding CXX object CMakeFiles/flashlight_lib_sequence_criterion.dir/bindings/python/flashlight/lib/sequence/_criterion.cpp.o\u001b[0m\n", "[ 90%] \u001b[32mBuilding CXX object CMakeFiles/flashlight_lib_text_dictionary.dir/bindings/python/flashlight/lib/text/_dictionary.cpp.o\u001b[0m\n", "[ 92%] \u001b[32mBuilding CXX object CMakeFiles/flashlight_lib_text_decoder.dir/bindings/python/flashlight/lib/text/_decoder.cpp.o\u001b[0m\n", "In file included from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/cast.h:16\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/attr.h:13\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/pybind11.h:43\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/flashlight/lib/text/_decoder.cpp:8\u001b[m\u001b[K:\n", "\u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/detail/internals.h:\u001b[m\u001b[K In function ‘\u001b[01m\u001b[Kpybind11::detail::internals& pybind11::detail::get_internals()\u001b[m\u001b[K’:\n", "\u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/detail/internals.h:194:28:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[K‘\u001b[01m\u001b[Kvoid PyEval_InitThreads()\u001b[m\u001b[K’ is deprecated [\u001b[01;35m\u001b[K-Wdeprecated-declarations\u001b[m\u001b[K]\n", " 194 | PyEval_InitThreads(\u001b[01;35m\u001b[K)\u001b[m\u001b[K;\n", " | \u001b[01;35m\u001b[K^\u001b[m\u001b[K\n", "In file included from \u001b[01m\u001b[K/usr/include/python3.10/Python.h:130\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/detail/common.h:112\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/pytypes.h:12\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/cast.h:13\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/attr.h:13\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/pybind11.h:43\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/flashlight/lib/text/_decoder.cpp:8\u001b[m\u001b[K:\n", "\u001b[01m\u001b[K/usr/include/python3.10/ceval.h:122:37:\u001b[m\u001b[K \u001b[01;36m\u001b[Knote: \u001b[m\u001b[Kdeclared here\n", " 122 | Py_DEPRECATED(3.9) PyAPI_FUNC(void) \u001b[01;36m\u001b[KPyEval_InitThreads\u001b[m\u001b[K(void);\n", " | \u001b[01;36m\u001b[K^~~~~~~~~~~~~~~~~~\u001b[m\u001b[K\n", "In file included from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/cast.h:16\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/attr.h:13\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/pybind11.h:43\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/flashlight/lib/text/_decoder.cpp:8\u001b[m\u001b[K:\n", "\u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/detail/internals.h:194:28:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[K‘\u001b[01m\u001b[Kvoid PyEval_InitThreads()\u001b[m\u001b[K’ is deprecated [\u001b[01;35m\u001b[K-Wdeprecated-declarations\u001b[m\u001b[K]\n", " 194 | PyEval_InitThreads(\u001b[01;35m\u001b[K)\u001b[m\u001b[K;\n", " | \u001b[01;35m\u001b[K^\u001b[m\u001b[K\n", "In file included from \u001b[01m\u001b[K/usr/include/python3.10/Python.h:130\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/detail/common.h:112\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/pytypes.h:12\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/cast.h:13\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/attr.h:13\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/pybind11.h:43\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/flashlight/lib/text/_decoder.cpp:8\u001b[m\u001b[K:\n", "\u001b[01m\u001b[K/usr/include/python3.10/ceval.h:122:37:\u001b[m\u001b[K \u001b[01;36m\u001b[Knote: \u001b[m\u001b[Kdeclared here\n", " 122 | Py_DEPRECATED(3.9) PyAPI_FUNC(void) \u001b[01;36m\u001b[KPyEval_InitThreads\u001b[m\u001b[K(void);\n", " | \u001b[01;36m\u001b[K^~~~~~~~~~~~~~~~~~\u001b[m\u001b[K\n", "In file included from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/cast.h:16\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/attr.h:13\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/pybind11.h:43\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/flashlight/lib/sequence/_criterion.cpp:8\u001b[m\u001b[K:\n", "\u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/detail/internals.h:\u001b[m\u001b[K In function ‘\u001b[01m\u001b[Kpybind11::detail::internals& pybind11::detail::get_internals()\u001b[m\u001b[K’:\n", "\u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/detail/internals.h:194:28:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[K‘\u001b[01m\u001b[Kvoid PyEval_InitThreads()\u001b[m\u001b[K’ is deprecated [\u001b[01;35m\u001b[K-Wdeprecated-declarations\u001b[m\u001b[K]\n", " 194 | PyEval_InitThreads(\u001b[01;35m\u001b[K)\u001b[m\u001b[K;\n", " | \u001b[01;35m\u001b[K^\u001b[m\u001b[K\n", "In file included from \u001b[01m\u001b[K/usr/include/python3.10/Python.h:130\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/detail/common.h:112\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/pytypes.h:12\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/cast.h:13\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/attr.h:13\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/pybind11.h:43\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/flashlight/lib/sequence/_criterion.cpp:8\u001b[m\u001b[K:\n", "\u001b[01m\u001b[K/usr/include/python3.10/ceval.h:122:37:\u001b[m\u001b[K \u001b[01;36m\u001b[Knote: \u001b[m\u001b[Kdeclared here\n", " 122 | Py_DEPRECATED(3.9) PyAPI_FUNC(void) \u001b[01;36m\u001b[KPyEval_InitThreads\u001b[m\u001b[K(void);\n", " | \u001b[01;36m\u001b[K^~~~~~~~~~~~~~~~~~\u001b[m\u001b[K\n", "In file included from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/cast.h:16\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/attr.h:13\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/pybind11.h:43\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/flashlight/lib/sequence/_criterion.cpp:8\u001b[m\u001b[K:\n", "\u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/detail/internals.h:194:28:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[K‘\u001b[01m\u001b[Kvoid PyEval_InitThreads()\u001b[m\u001b[K’ is deprecated [\u001b[01;35m\u001b[K-Wdeprecated-declarations\u001b[m\u001b[K]\n", " 194 | PyEval_InitThreads(\u001b[01;35m\u001b[K)\u001b[m\u001b[K;\n", " | \u001b[01;35m\u001b[K^\u001b[m\u001b[K\n", "In file included from \u001b[01m\u001b[K/usr/include/python3.10/Python.h:130\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/detail/common.h:112\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/pytypes.h:12\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/cast.h:13\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/attr.h:13\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/pybind11.h:43\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/flashlight/lib/sequence/_criterion.cpp:8\u001b[m\u001b[K:\n", "\u001b[01m\u001b[K/usr/include/python3.10/ceval.h:122:37:\u001b[m\u001b[K \u001b[01;36m\u001b[Knote: \u001b[m\u001b[Kdeclared here\n", " 122 | Py_DEPRECATED(3.9) PyAPI_FUNC(void) \u001b[01;36m\u001b[KPyEval_InitThreads\u001b[m\u001b[K(void);\n", " | \u001b[01;36m\u001b[K^~~~~~~~~~~~~~~~~~\u001b[m\u001b[K\n", "In file included from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/cast.h:16\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/attr.h:13\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/pybind11.h:43\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/flashlight/lib/text/_dictionary.cpp:8\u001b[m\u001b[K:\n", "\u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/detail/internals.h:\u001b[m\u001b[K In function ‘\u001b[01m\u001b[Kpybind11::detail::internals& pybind11::detail::get_internals()\u001b[m\u001b[K’:\n", "\u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/detail/internals.h:194:28:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[K‘\u001b[01m\u001b[Kvoid PyEval_InitThreads()\u001b[m\u001b[K’ is deprecated [\u001b[01;35m\u001b[K-Wdeprecated-declarations\u001b[m\u001b[K]\n", " 194 | PyEval_InitThreads(\u001b[01;35m\u001b[K)\u001b[m\u001b[K;\n", " | \u001b[01;35m\u001b[K^\u001b[m\u001b[K\n", "In file included from \u001b[01m\u001b[K/usr/include/python3.10/Python.h:130\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/detail/common.h:112\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/pytypes.h:12\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/cast.h:13\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/attr.h:13\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/pybind11.h:43\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/flashlight/lib/text/_dictionary.cpp:8\u001b[m\u001b[K:\n", "\u001b[01m\u001b[K/usr/include/python3.10/ceval.h:122:37:\u001b[m\u001b[K \u001b[01;36m\u001b[Knote: \u001b[m\u001b[Kdeclared here\n", " 122 | Py_DEPRECATED(3.9) PyAPI_FUNC(void) \u001b[01;36m\u001b[KPyEval_InitThreads\u001b[m\u001b[K(void);\n", " | \u001b[01;36m\u001b[K^~~~~~~~~~~~~~~~~~\u001b[m\u001b[K\n", "In file included from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/cast.h:16\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/attr.h:13\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/pybind11.h:43\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/flashlight/lib/text/_dictionary.cpp:8\u001b[m\u001b[K:\n", "\u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/detail/internals.h:194:28:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[K‘\u001b[01m\u001b[Kvoid PyEval_InitThreads()\u001b[m\u001b[K’ is deprecated [\u001b[01;35m\u001b[K-Wdeprecated-declarations\u001b[m\u001b[K]\n", " 194 | PyEval_InitThreads(\u001b[01;35m\u001b[K)\u001b[m\u001b[K;\n", " | \u001b[01;35m\u001b[K^\u001b[m\u001b[K\n", "In file included from \u001b[01m\u001b[K/usr/include/python3.10/Python.h:130\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/detail/common.h:112\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/pytypes.h:12\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/cast.h:13\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/attr.h:13\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/pybind11.h:43\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/flashlight/lib/text/_dictionary.cpp:8\u001b[m\u001b[K:\n", "\u001b[01m\u001b[K/usr/include/python3.10/ceval.h:122:37:\u001b[m\u001b[K \u001b[01;36m\u001b[Knote: \u001b[m\u001b[Kdeclared here\n", " 122 | Py_DEPRECATED(3.9) PyAPI_FUNC(void) \u001b[01;36m\u001b[KPyEval_InitThreads\u001b[m\u001b[K(void);\n", " | \u001b[01;36m\u001b[K^~~~~~~~~~~~~~~~~~\u001b[m\u001b[K\n", "In file included from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/cast.h:16\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/attr.h:13\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/pybind11.h:43\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/flashlight/lib/audio/_feature.cpp:8\u001b[m\u001b[K:\n", "\u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/detail/internals.h:\u001b[m\u001b[K In function ‘\u001b[01m\u001b[Kpybind11::detail::internals& pybind11::detail::get_internals()\u001b[m\u001b[K’:\n", "\u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/detail/internals.h:194:28:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[K‘\u001b[01m\u001b[Kvoid PyEval_InitThreads()\u001b[m\u001b[K’ is deprecated [\u001b[01;35m\u001b[K-Wdeprecated-declarations\u001b[m\u001b[K]\n", " 194 | PyEval_InitThreads(\u001b[01;35m\u001b[K)\u001b[m\u001b[K;\n", " | \u001b[01;35m\u001b[K^\u001b[m\u001b[K\n", "In file included from \u001b[01m\u001b[K/usr/include/python3.10/Python.h:130\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/detail/common.h:112\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/pytypes.h:12\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/cast.h:13\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/attr.h:13\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/pybind11.h:43\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/flashlight/lib/audio/_feature.cpp:8\u001b[m\u001b[K:\n", "\u001b[01m\u001b[K/usr/include/python3.10/ceval.h:122:37:\u001b[m\u001b[K \u001b[01;36m\u001b[Knote: \u001b[m\u001b[Kdeclared here\n", " 122 | Py_DEPRECATED(3.9) PyAPI_FUNC(void) \u001b[01;36m\u001b[KPyEval_InitThreads\u001b[m\u001b[K(void);\n", " | \u001b[01;36m\u001b[K^~~~~~~~~~~~~~~~~~\u001b[m\u001b[K\n", "In file included from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/cast.h:16\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/attr.h:13\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/pybind11.h:43\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/flashlight/lib/audio/_feature.cpp:8\u001b[m\u001b[K:\n", "\u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/detail/internals.h:194:28:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[K‘\u001b[01m\u001b[Kvoid PyEval_InitThreads()\u001b[m\u001b[K’ is deprecated [\u001b[01;35m\u001b[K-Wdeprecated-declarations\u001b[m\u001b[K]\n", " 194 | PyEval_InitThreads(\u001b[01;35m\u001b[K)\u001b[m\u001b[K;\n", " | \u001b[01;35m\u001b[K^\u001b[m\u001b[K\n", "In file included from \u001b[01m\u001b[K/usr/include/python3.10/Python.h:130\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/detail/common.h:112\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/pytypes.h:12\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/cast.h:13\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/attr.h:13\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310/pybind11/src/pybind11/include/pybind11/pybind11.h:43\u001b[m\u001b[K,\n", " from \u001b[01m\u001b[K/content/fairseq/flashlight/bindings/python/flashlight/lib/audio/_feature.cpp:8\u001b[m\u001b[K:\n", "\u001b[01m\u001b[K/usr/include/python3.10/ceval.h:122:37:\u001b[m\u001b[K \u001b[01;36m\u001b[Knote: \u001b[m\u001b[Kdeclared here\n", " 122 | Py_DEPRECATED(3.9) PyAPI_FUNC(void) \u001b[01;36m\u001b[KPyEval_InitThreads\u001b[m\u001b[K(void);\n", " | \u001b[01;36m\u001b[K^~~~~~~~~~~~~~~~~~\u001b[m\u001b[K\n", "[ 94%] \u001b[32m\u001b[1mLinking CXX shared module /content/fairseq/flashlight/bindings/python/build/lib.linux-x86_64-cpython-310/flashlight/lib/text/flashlight_lib_text_dictionary.cpython-310-x86_64-linux-gnu.so\u001b[0m\n", "[ 96%] \u001b[32m\u001b[1mLinking CXX shared module /content/fairseq/flashlight/bindings/python/build/lib.linux-x86_64-cpython-310/flashlight/lib/sequence/flashlight_lib_sequence_criterion.cpython-310-x86_64-linux-gnu.so\u001b[0m\n", "[ 98%] \u001b[32m\u001b[1mLinking CXX shared module /content/fairseq/flashlight/bindings/python/build/lib.linux-x86_64-cpython-310/flashlight/lib/audio/flashlight_lib_audio_feature.cpython-310-x86_64-linux-gnu.so\u001b[0m\n", "[ 98%] Built target flashlight_lib_text_dictionary\n", "[100%] \u001b[32m\u001b[1mLinking CXX shared module /content/fairseq/flashlight/bindings/python/build/lib.linux-x86_64-cpython-310/flashlight/lib/text/flashlight_lib_text_decoder.cpython-310-x86_64-linux-gnu.so\u001b[0m\n", "[100%] Built target flashlight_lib_sequence_criterion\n", "[100%] Built target flashlight_lib_audio_feature\n", "[100%] Built target flashlight_lib_text_decoder\n", "-- -rdynamic supported.\n", "-- CUDA found (library: /usr/local/cuda/lib64/libcudart_static.a;Threads::Threads;dl;/usr/lib/x86_64-linux-gnu/librt.so include: /usr/local/cuda/include)\n", "-- CUDA architecture flags: -gencodearch=compute_35,code=sm_35-gencodearch=compute_50,code=sm_50-gencodearch=compute_52,code=sm_52-gencodearch=compute_60,code=sm_60-gencodearch=compute_61,code=sm_61-gencodearch=compute_70,code=sm_70-gencodearch=compute_75,code=sm_75-gencodearch=compute_80,code=sm_80-gencodearch=compute_80,code=compute_80\n", "-- Will build flashlight libraries.\n", "-- MKL_THREADING = OMP\n", "-- Checking for [mkl_intel_lp64 - mkl_gnu_thread - mkl_core - gomp - pthread - m - dl]\n", "-- Library mkl_intel_lp64: /opt/intel/mkl/lib/intel64/libmkl_intel_lp64.so\n", "-- Library mkl_gnu_thread: /opt/intel/mkl/lib/intel64/libmkl_gnu_thread.so\n", "-- Library mkl_core: /opt/intel/mkl/lib/intel64/libmkl_core.so\n", "-- Library gomp: -fopenmp\n", "-- Library pthread: /usr/lib/x86_64-linux-gnu/libpthread.so\n", "-- Library m: /usr/lib/x86_64-linux-gnu/libm.so\n", "-- Library dl: /usr/lib/x86_64-linux-gnu/libdl.so\n", "-- MKL library found\n", "-- CBLAS found (include: /opt/intel/mkl/include, library: /opt/intel/mkl/lib/intel64/libmkl_intel_lp64.so;/opt/intel/mkl/lib/intel64/libmkl_gnu_thread.so;/opt/intel/mkl/lib/intel64/libmkl_core.so;-fopenmp;/usr/lib/x86_64-linux-gnu/libpthread.so;/usr/lib/x86_64-linux-gnu/libm.so;/usr/lib/x86_64-linux-gnu/libdl.so)\n", "-- Could NOT find FFTW3 (missing: FFTW3_DIR)\n", "-- FindFFTW using pkgconfig: FOUND=1 LIBRARIES=fftw3 LIBRARY_DIRS=/usr/lib/x86_64-linux-gnu LIBDIR=/usr/lib/x86_64-linux-gnu LINK_LIBRARIES=/usr/lib/x86_64-linux-gnu/libfftw3.so\n", "-- FindFTTW using pkgconfig: INCLUDE_DIRS= INCLUDEDIR=/usr/include\n", "-- FFTW found\n", "-- Looking for KenLM\n", "-- Using kenlm library found in /usr/local/lib/libkenlm.a\n", "-- Using kenlm utils library found in /usr/local/lib/libkenlm_util.a\n", "-- kenlm model.hh found in /usr/local/include/kenlm/lm\n", "-- Found kenlm (include: /usr/local/include, library: /usr/local/lib/libkenlm.a;/usr/local/lib/libkenlm_util.a;/usr/lib/x86_64-linux-gnu/liblzma.so;/usr/lib/x86_64-linux-gnu/libbz2.so;/usr/lib/x86_64-linux-gnu/libz.so)\n", "-- Configuring done\n", "\u001b[33mCMake Warning (dev) in CMakeLists.txt:\n", " Policy CMP0104 is not set: CMAKE_CUDA_ARCHITECTURES now detected for NVCC,\n", " empty CUDA_ARCHITECTURES not allowed. Run \"cmake --help-policy CMP0104\"\n", " for policy details. Use the cmake_policy command to set the policy and\n", " suppress this warning.\n", "\n", " CUDA_ARCHITECTURES is empty for target \"fl-libraries\".\n", "This warning is for project developers. Use -Wno-dev to suppress it.\n", "\u001b[0m\n", "-- Generating done\n", "-- Build files have been written to: /content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310\n", "[ 1%] \u001b[34m\u001b[1mPerforming update step for 'pybind11'\u001b[0m\n", "[ 3%] \u001b[34m\u001b[1mNo patch step for 'pybind11'\u001b[0m\n", "[ 5%] \u001b[34m\u001b[1mNo configure step for 'pybind11'\u001b[0m\n", "[ 7%] \u001b[34m\u001b[1mNo build step for 'pybind11'\u001b[0m\n", "[ 9%] \u001b[34m\u001b[1mNo install step for 'pybind11'\u001b[0m\n", "[ 11%] \u001b[34m\u001b[1mCompleted 'pybind11'\u001b[0m\n", "[ 15%] Built target pybind11\n", "[ 84%] Built target fl-libraries\n", "[ 88%] Built target flashlight_lib_sequence_criterion\n", "[ 92%] Built target flashlight_lib_text_decoder\n", "[ 96%] Built target flashlight_lib_audio_feature\n", "[100%] Built target flashlight_lib_text_dictionary\n", "-- -rdynamic supported.\n", "-- CUDA found (library: /usr/local/cuda/lib64/libcudart_static.a;Threads::Threads;dl;/usr/lib/x86_64-linux-gnu/librt.so include: /usr/local/cuda/include)\n", "-- CUDA architecture flags: -gencodearch=compute_35,code=sm_35-gencodearch=compute_50,code=sm_50-gencodearch=compute_52,code=sm_52-gencodearch=compute_60,code=sm_60-gencodearch=compute_61,code=sm_61-gencodearch=compute_70,code=sm_70-gencodearch=compute_75,code=sm_75-gencodearch=compute_80,code=sm_80-gencodearch=compute_80,code=compute_80\n", "-- Will build flashlight libraries.\n", "-- MKL_THREADING = OMP\n", "-- Checking for [mkl_intel_lp64 - mkl_gnu_thread - mkl_core - gomp - pthread - m - dl]\n", "-- Library mkl_intel_lp64: /opt/intel/mkl/lib/intel64/libmkl_intel_lp64.so\n", "-- Library mkl_gnu_thread: /opt/intel/mkl/lib/intel64/libmkl_gnu_thread.so\n", "-- Library mkl_core: /opt/intel/mkl/lib/intel64/libmkl_core.so\n", "-- Library gomp: -fopenmp\n", "-- Library pthread: /usr/lib/x86_64-linux-gnu/libpthread.so\n", "-- Library m: /usr/lib/x86_64-linux-gnu/libm.so\n", "-- Library dl: /usr/lib/x86_64-linux-gnu/libdl.so\n", "-- MKL library found\n", "-- CBLAS found (include: /opt/intel/mkl/include, library: /opt/intel/mkl/lib/intel64/libmkl_intel_lp64.so;/opt/intel/mkl/lib/intel64/libmkl_gnu_thread.so;/opt/intel/mkl/lib/intel64/libmkl_core.so;-fopenmp;/usr/lib/x86_64-linux-gnu/libpthread.so;/usr/lib/x86_64-linux-gnu/libm.so;/usr/lib/x86_64-linux-gnu/libdl.so)\n", "-- Could NOT find FFTW3 (missing: FFTW3_DIR)\n", "-- FindFFTW using pkgconfig: FOUND=1 LIBRARIES=fftw3 LIBRARY_DIRS=/usr/lib/x86_64-linux-gnu LIBDIR=/usr/lib/x86_64-linux-gnu LINK_LIBRARIES=/usr/lib/x86_64-linux-gnu/libfftw3.so\n", "-- FindFTTW using pkgconfig: INCLUDE_DIRS= INCLUDEDIR=/usr/include\n", "-- FFTW found\n", "-- Looking for KenLM\n", "-- Using kenlm library found in /usr/local/lib/libkenlm.a\n", "-- Using kenlm utils library found in /usr/local/lib/libkenlm_util.a\n", "-- kenlm model.hh found in /usr/local/include/kenlm/lm\n", "-- Found kenlm (include: /usr/local/include, library: /usr/local/lib/libkenlm.a;/usr/local/lib/libkenlm_util.a;/usr/lib/x86_64-linux-gnu/liblzma.so;/usr/lib/x86_64-linux-gnu/libbz2.so;/usr/lib/x86_64-linux-gnu/libz.so)\n", "-- Configuring done\n", "\u001b[33mCMake Warning (dev) in CMakeLists.txt:\n", " Policy CMP0104 is not set: CMAKE_CUDA_ARCHITECTURES now detected for NVCC,\n", " empty CUDA_ARCHITECTURES not allowed. Run \"cmake --help-policy CMP0104\"\n", " for policy details. Use the cmake_policy command to set the policy and\n", " suppress this warning.\n", "\n", " CUDA_ARCHITECTURES is empty for target \"fl-libraries\".\n", "This warning is for project developers. Use -Wno-dev to suppress it.\n", "\u001b[0m\n", "-- Generating done\n", "-- Build files have been written to: /content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310\n", "[ 1%] \u001b[34m\u001b[1mPerforming update step for 'pybind11'\u001b[0m\n", "[ 3%] \u001b[34m\u001b[1mNo patch step for 'pybind11'\u001b[0m\n", "[ 5%] \u001b[34m\u001b[1mNo configure step for 'pybind11'\u001b[0m\n", "[ 7%] \u001b[34m\u001b[1mNo build step for 'pybind11'\u001b[0m\n", "[ 9%] \u001b[34m\u001b[1mNo install step for 'pybind11'\u001b[0m\n", "[ 11%] \u001b[34m\u001b[1mCompleted 'pybind11'\u001b[0m\n", "[ 15%] Built target pybind11\n", "[ 84%] Built target fl-libraries\n", "[ 88%] Built target flashlight_lib_sequence_criterion\n", "[ 92%] Built target flashlight_lib_audio_feature\n", "[ 96%] Built target flashlight_lib_text_decoder\n", "[100%] Built target flashlight_lib_text_dictionary\n", "-- -rdynamic supported.\n", "-- CUDA found (library: /usr/local/cuda/lib64/libcudart_static.a;Threads::Threads;dl;/usr/lib/x86_64-linux-gnu/librt.so include: /usr/local/cuda/include)\n", "-- CUDA architecture flags: -gencodearch=compute_35,code=sm_35-gencodearch=compute_50,code=sm_50-gencodearch=compute_52,code=sm_52-gencodearch=compute_60,code=sm_60-gencodearch=compute_61,code=sm_61-gencodearch=compute_70,code=sm_70-gencodearch=compute_75,code=sm_75-gencodearch=compute_80,code=sm_80-gencodearch=compute_80,code=compute_80\n", "-- Will build flashlight libraries.\n", "-- MKL_THREADING = OMP\n", "-- Checking for [mkl_intel_lp64 - mkl_gnu_thread - mkl_core - gomp - pthread - m - dl]\n", "-- Library mkl_intel_lp64: /opt/intel/mkl/lib/intel64/libmkl_intel_lp64.so\n", "-- Library mkl_gnu_thread: /opt/intel/mkl/lib/intel64/libmkl_gnu_thread.so\n", "-- Library mkl_core: /opt/intel/mkl/lib/intel64/libmkl_core.so\n", "-- Library gomp: -fopenmp\n", "-- Library pthread: /usr/lib/x86_64-linux-gnu/libpthread.so\n", "-- Library m: /usr/lib/x86_64-linux-gnu/libm.so\n", "-- Library dl: /usr/lib/x86_64-linux-gnu/libdl.so\n", "-- MKL library found\n", "-- CBLAS found (include: /opt/intel/mkl/include, library: /opt/intel/mkl/lib/intel64/libmkl_intel_lp64.so;/opt/intel/mkl/lib/intel64/libmkl_gnu_thread.so;/opt/intel/mkl/lib/intel64/libmkl_core.so;-fopenmp;/usr/lib/x86_64-linux-gnu/libpthread.so;/usr/lib/x86_64-linux-gnu/libm.so;/usr/lib/x86_64-linux-gnu/libdl.so)\n", "-- Could NOT find FFTW3 (missing: FFTW3_DIR)\n", "-- FindFFTW using pkgconfig: FOUND=1 LIBRARIES=fftw3 LIBRARY_DIRS=/usr/lib/x86_64-linux-gnu LIBDIR=/usr/lib/x86_64-linux-gnu LINK_LIBRARIES=/usr/lib/x86_64-linux-gnu/libfftw3.so\n", "-- FindFTTW using pkgconfig: INCLUDE_DIRS= INCLUDEDIR=/usr/include\n", "-- FFTW found\n", "-- Looking for KenLM\n", "-- Using kenlm library found in /usr/local/lib/libkenlm.a\n", "-- Using kenlm utils library found in /usr/local/lib/libkenlm_util.a\n", "-- kenlm model.hh found in /usr/local/include/kenlm/lm\n", "-- Found kenlm (include: /usr/local/include, library: /usr/local/lib/libkenlm.a;/usr/local/lib/libkenlm_util.a;/usr/lib/x86_64-linux-gnu/liblzma.so;/usr/lib/x86_64-linux-gnu/libbz2.so;/usr/lib/x86_64-linux-gnu/libz.so)\n", "-- Configuring done\n", "\u001b[33mCMake Warning (dev) in CMakeLists.txt:\n", " Policy CMP0104 is not set: CMAKE_CUDA_ARCHITECTURES now detected for NVCC,\n", " empty CUDA_ARCHITECTURES not allowed. Run \"cmake --help-policy CMP0104\"\n", " for policy details. Use the cmake_policy command to set the policy and\n", " suppress this warning.\n", "\n", " CUDA_ARCHITECTURES is empty for target \"fl-libraries\".\n", "This warning is for project developers. Use -Wno-dev to suppress it.\n", "\u001b[0m\n", "-- Generating done\n", "-- Build files have been written to: /content/fairseq/flashlight/bindings/python/build/temp.linux-x86_64-cpython-310\n", "[ 1%] \u001b[34m\u001b[1mPerforming update step for 'pybind11'\u001b[0m\n", "[ 3%] \u001b[34m\u001b[1mNo patch step for 'pybind11'\u001b[0m\n", "[ 5%] \u001b[34m\u001b[1mNo configure step for 'pybind11'\u001b[0m\n", "[ 7%] \u001b[34m\u001b[1mNo build step for 'pybind11'\u001b[0m\n", "[ 9%] \u001b[34m\u001b[1mNo install step for 'pybind11'\u001b[0m\n", "[ 11%] \u001b[34m\u001b[1mCompleted 'pybind11'\u001b[0m\n", "[ 81%] Built target fl-libraries\n", "[ 84%] Built target pybind11\n", "[ 88%] Built target flashlight_lib_sequence_criterion\n", "[ 92%] Built target flashlight_lib_audio_feature\n", "[ 96%] Built target flashlight_lib_text_decoder\n", "[100%] Built target flashlight_lib_text_dictionary\n", "creating build/bdist.linux-x86_64\n", "creating build/bdist.linux-x86_64/egg\n", "copying build/lib.linux-x86_64-cpython-310/libfl-libraries.so -> build/bdist.linux-x86_64/egg\n", "copying build/lib.linux-x86_64-cpython-310/libfl-libraries.so.0 -> build/bdist.linux-x86_64/egg\n", "copying build/lib.linux-x86_64-cpython-310/libfl-libraries.so.0.3 -> build/bdist.linux-x86_64/egg\n", "creating build/bdist.linux-x86_64/egg/flashlight\n", "creating build/bdist.linux-x86_64/egg/flashlight/lib\n", "creating build/bdist.linux-x86_64/egg/flashlight/lib/sequence\n", "copying build/lib.linux-x86_64-cpython-310/flashlight/lib/sequence/flashlight_lib_sequence_criterion.cpython-310-x86_64-linux-gnu.so -> build/bdist.linux-x86_64/egg/flashlight/lib/sequence\n", "copying build/lib.linux-x86_64-cpython-310/flashlight/lib/sequence/criterion_torch.py -> build/bdist.linux-x86_64/egg/flashlight/lib/sequence\n", "copying build/lib.linux-x86_64-cpython-310/flashlight/lib/sequence/criterion.py -> build/bdist.linux-x86_64/egg/flashlight/lib/sequence\n", "creating build/bdist.linux-x86_64/egg/flashlight/lib/audio\n", "copying build/lib.linux-x86_64-cpython-310/flashlight/lib/audio/feature.py -> build/bdist.linux-x86_64/egg/flashlight/lib/audio\n", "copying build/lib.linux-x86_64-cpython-310/flashlight/lib/audio/flashlight_lib_audio_feature.cpython-310-x86_64-linux-gnu.so -> build/bdist.linux-x86_64/egg/flashlight/lib/audio\n", "creating build/bdist.linux-x86_64/egg/flashlight/lib/text\n", "copying build/lib.linux-x86_64-cpython-310/flashlight/lib/text/flashlight_lib_text_dictionary.cpython-310-x86_64-linux-gnu.so -> build/bdist.linux-x86_64/egg/flashlight/lib/text\n", "copying build/lib.linux-x86_64-cpython-310/flashlight/lib/text/flashlight_lib_text_decoder.cpython-310-x86_64-linux-gnu.so -> build/bdist.linux-x86_64/egg/flashlight/lib/text\n", "copying build/lib.linux-x86_64-cpython-310/flashlight/lib/text/dictionary.py -> build/bdist.linux-x86_64/egg/flashlight/lib/text\n", "copying build/lib.linux-x86_64-cpython-310/flashlight/lib/text/decoder.py -> build/bdist.linux-x86_64/egg/flashlight/lib/text\n", "copying build/lib.linux-x86_64-cpython-310/flashlight/lib/__init__.py -> build/bdist.linux-x86_64/egg/flashlight/lib\n", "byte-compiling build/bdist.linux-x86_64/egg/flashlight/lib/sequence/criterion_torch.py to criterion_torch.cpython-310.pyc\n", "byte-compiling build/bdist.linux-x86_64/egg/flashlight/lib/sequence/criterion.py to criterion.cpython-310.pyc\n", "byte-compiling build/bdist.linux-x86_64/egg/flashlight/lib/audio/feature.py to feature.cpython-310.pyc\n", "byte-compiling build/bdist.linux-x86_64/egg/flashlight/lib/text/dictionary.py to dictionary.cpython-310.pyc\n", "byte-compiling build/bdist.linux-x86_64/egg/flashlight/lib/text/decoder.py to decoder.cpython-310.pyc\n", "byte-compiling build/bdist.linux-x86_64/egg/flashlight/lib/__init__.py to __init__.cpython-310.pyc\n", "creating build/bdist.linux-x86_64/egg/EGG-INFO\n", "copying flashlight.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO\n", "copying flashlight.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO\n", "copying flashlight.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO\n", "copying flashlight.egg-info/not-zip-safe -> build/bdist.linux-x86_64/egg/EGG-INFO\n", "copying flashlight.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO\n", "writing build/bdist.linux-x86_64/egg/EGG-INFO/native_libs.txt\n", "creating dist\n", "creating 'dist/flashlight-1.0.0-py3.10-linux-x86_64.egg' and adding 'build/bdist.linux-x86_64/egg' to it\n", "removing 'build/bdist.linux-x86_64/egg' (and everything under it)\n", "Processing flashlight-1.0.0-py3.10-linux-x86_64.egg\n", "creating /usr/local/lib/python3.10/dist-packages/flashlight-1.0.0-py3.10-linux-x86_64.egg\n", "Extracting flashlight-1.0.0-py3.10-linux-x86_64.egg to /usr/local/lib/python3.10/dist-packages\n", "Adding flashlight 1.0.0 to easy-install.pth file\n", "\n", "Installed /usr/local/lib/python3.10/dist-packages/flashlight-1.0.0-py3.10-linux-x86_64.egg\n", "Processing dependencies for flashlight==1.0.0\n", "Finished processing dependencies for flashlight==1.0.0\n", "/content/fairseq\n" ] } ] }, { "cell_type": "markdown", "source": [ "Next, we download an audio file from [People's speech](https://huggingface.co/datasets/MLCommons/peoples_speech) data. We will the audio sample from their 'dirty' subset which will be more challenging for the ASR model. " ], "metadata": { "id": "1ejR32MNbyf9" } }, { "cell_type": "code", "source": [ "!wget -O ./audio_samples/tmp.wav 'https://datasets-server.huggingface.co/assets/MLCommons/peoples_speech/--/dirty/train/0/audio/audio.wav'\n", "!ffmpeg -y -i ./audio_samples/tmp.wav -ar 16000 ./audio_samples/audio_noisy.wav\n", "\n" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "LCB_t3-fbcM6", "outputId": "3a61229b-e4c9-4055-ac1c-c0f8159a3f00" }, "execution_count": 12, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "--2023-05-26 00:26:41-- https://datasets-server.huggingface.co/assets/MLCommons/peoples_speech/--/dirty/train/0/audio/audio.wav\n", "Resolving datasets-server.huggingface.co (datasets-server.huggingface.co)... 34.200.186.24, 3.216.183.114, 44.197.252.161, ...\n", "Connecting to datasets-server.huggingface.co (datasets-server.huggingface.co)|34.200.186.24|:443... connected.\n", "HTTP request sent, awaiting response... 200 OK\n", "Length: 386924 (378K) [application/octet-stream]\n", "Saving to: ‘./audio_samples/tmp.wav’\n", "\n", "./audio_samples/tmp 100%[===================>] 377.86K 1.07MB/s in 0.3s \n", "\n", "2023-05-26 00:26:42 (1.07 MB/s) - ‘./audio_samples/tmp.wav’ saved [386924/386924]\n", "\n", "ffmpeg version 4.2.7-0ubuntu0.1 Copyright (c) 2000-2022 the FFmpeg developers\n", " built with gcc 9 (Ubuntu 9.4.0-1ubuntu1~20.04.1)\n", " configuration: --prefix=/usr --extra-version=0ubuntu0.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared\n", " libavutil 56. 31.100 / 56. 31.100\n", " libavcodec 58. 54.100 / 58. 54.100\n", " libavformat 58. 29.100 / 58. 29.100\n", " libavdevice 58. 8.100 / 58. 8.100\n", " libavfilter 7. 57.100 / 7. 57.100\n", " libavresample 4. 0. 0 / 4. 0. 0\n", " libswscale 5. 5.100 / 5. 5.100\n", " libswresample 3. 5.100 / 3. 5.100\n", " libpostproc 55. 5.100 / 55. 5.100\n", "\u001b[0;33mGuessed Channel Layout for Input Stream #0.0 : mono\n", "\u001b[0mInput #0, wav, from './audio_samples/tmp.wav':\n", " Duration: 00:00:12.09, bitrate: 256 kb/s\n", " Stream #0:0: Audio: pcm_s16le ([1][0][0][0] / 0x0001), 16000 Hz, mono, s16, 256 kb/s\n", "Stream mapping:\n", " Stream #0:0 -> #0:0 (pcm_s16le (native) -> pcm_s16le (native))\n", "Press [q] to stop, [?] for help\n", "Output #0, wav, to './audio_samples/audio_noisy.wav':\n", " Metadata:\n", " ISFT : Lavf58.29.100\n", " Stream #0:0: Audio: pcm_s16le ([1][0][0][0] / 0x0001), 16000 Hz, mono, s16, 256 kb/s\n", " Metadata:\n", " encoder : Lavc58.54.100 pcm_s16le\n", "size= 378kB time=00:00:12.09 bitrate= 256.1kbits/s speed= 846x \n", "video:0kB audio:378kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.020161%\n", "Trancript: limiting emotions that we experience pain in our childhood which stop us from living our life just open freedom i mean trust\n" ] } ] }, { "cell_type": "markdown", "source": [ "Let's listen to the audio file \n" ], "metadata": { "id": "Iq_IWIpqc7hK" } }, { "cell_type": "code", "source": [ "import IPython\n", "IPython.display.display(IPython.display.Audio(\"./audio_samples/audio_noisy.wav\"))\n", "print(\"Trancript: limiting emotions that we experience mainly in our childhood which stop us from living our life just open freedom i mean trust and\")" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 93 }, "id": "IrqHRS0Sc_Oo", "outputId": "a1ac89d9-30c0-4889-c3bc-1fa336cf0c79" }, "execution_count": 16, "outputs": [ { "output_type": "display_data", "data": { "text/plain": [ "" ], "text/html": [ "\n", " \n", " " ] }, "metadata": {} }, { "output_type": "stream", "name": "stdout", "text": [ "Trancript: limiting emotions that we experience mainly in our childhood which stop us from living our life just open freedom i mean trust and\n" ] } ] }, { "cell_type": "markdown", "source": [ "Run inference with both greedy decoding and LM decoding" ], "metadata": { "id": "ZdVDTW_yduyQ" } }, { "cell_type": "code", "source": [ "import os\n", "\n", "os.environ[\"TMPDIR\"] = '/content/temp_dir'\n", "os.environ[\"PYTHONPATH\"] = \".\"\n", "os.environ[\"PREFIX\"] = \"INFER\"\n", "os.environ[\"HYDRA_FULL_ERROR\"] = \"1\"\n", "os.environ[\"USER\"] = \"micro\"\n", "\n", "print(\"======= WITHOUT LM DECODING=======\")\n", "\n", "!python examples/mms/asr/infer/mms_infer.py --model \"/content/fairseq/models_new/mms1b_fl102.pt\" --lang \"eng\" --audio \"/content/fairseq/audio_samples/audio.wav\" \"/content/fairseq/audio_samples/audio_noisy.wav\"\n", "\n", "print(\"\\n\\n\\n======= WITH LM DECODING=======\")\n", "\n", "# Note that the lmweight, wordscore needs to tuned for each LM \n", "# Using the same values may not be optimal\n", "decoding_cmds = \"\"\"\n", "decoding.type=kenlm \n", "decoding.beam=500 \n", "decoding.beamsizetoken=50 \n", "decoding.lmweight=2.69\n", "decoding.wordscore=2.8\n", "decoding.lmpath=/content/lmdecode/lm_common_crawl_small_4gram_prun0-6-15_200kvocab.bin\n", "decoding.lexicon=/content/lmdecode/lexicon.txt\n", "\"\"\".replace(\"\\n\", \" \")\n", "!python examples/mms/asr/infer/mms_infer.py --model \"/content/fairseq/models_new/mms1b_fl102.pt\" --lang \"eng\" --audio \"/content/fairseq/audio_samples/audio.wav\" \"/content/fairseq/audio_samples/audio_noisy.wav\" \\\n", " --extra-infer-args '{decoding_cmds}'\n" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "JlKzKBIlZqKq", "outputId": "94da50cc-2672-418b-a941-24ea0db0339b" }, "execution_count": 39, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "======= WITHOUT LM DECODING=======\n", ">>> preparing tmp manifest dir ...\n", ">>> loading model & running inference ...\n", "2023-05-26 01:01:58.415006: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.\n", "To enable the following instructions: AVX2 AVX512F FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.\n", "2023-05-26 01:02:00.361210: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n", "===============\n", "Input: /content/fairseq/audio_samples/audio.wav\n", "Output: a tornado is a spinning colum of very low-pressure air which sucks it surrounding air inward and upward\n", "===============\n", "Input: /content/fairseq/audio_samples/audio_noisy.wav\n", "Output: limiting emotions that weexperienced mainly in our childhood which stop us from living our lives in just open freedom and interust and\n", "======= WITH LM DECODING=======\n", ">>> preparing tmp manifest dir ...\n", ">>> loading model & running inference ...\n", "2023-05-26 01:03:50.066828: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.\n", "To enable the following instructions: AVX2 AVX512F FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.\n", "2023-05-26 01:03:52.190710: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n", "===============\n", "Input: /content/fairseq/audio_samples/audio.wav\n", "Output: a tornado is a spinning column of very low pressure air which sucks at surrounding air inward and upward\n", "===============\n", "Input: /content/fairseq/audio_samples/audio_noisy.wav\n", "Output: limiting emotions that we experience mainly in our childhood which stop us from living our lives in just open freedom and interest and\n" ] } ] }, { "cell_type": "code", "source": [], "metadata": { "id": "dJUXy2dye8sH" }, "execution_count": null, "outputs": [] } ] }