{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Export bluesky data to CSV\n", "\n", "Export bluesky experiment data from databroker to CSV files. Each data stream is written to its own file.\n", "\n", "As a bonus, it is shown how to write all of a run's data to a single JSON file." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "First, identify the experiment data to be exported. Show what catalogs are available." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['class_2021_03', 'class_example_data']" ] }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import databroker\n", "list(databroker.catalog)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Pick from the `class_example_data` catalog." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "59" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "cat = databroker.catalog[\"class_example_data\"]\n", "len(cat)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Just show a few of the most recent runs." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "catalog: class_example_data\n", " scan_id time plan_name detectors\n", "0 90 2021-03-06 14:16:41 scan [noisy]\n", "1 89 2021-03-06 14:15:35 scan [noisy]\n", "2 88 2021-03-06 14:14:45 scan [noisy]\n", "3 87 2021-03-06 14:13:44 scan [noisy]\n", "4 86 2021-03-06 14:10:46 rel_scan [noisy]\n", "5 85 2021-03-06 14:10:43 rel_scan [noisy]\n", "6 84 2021-03-06 14:10:37 rel_scan [noisy]\n", "7 83 2021-03-06 14:10:19 rel_scan [noisy]\n", "8 82 2021-03-03 10:01:32 count [adsimdet]\n" ] } ], "source": [ "from apstools.utils import *\n", "listruns(num=9)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Check what is in #85." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "BlueskyRun\n", " uid='b4e4cbbb-a4b1-4146-aea5-ba8bc7ad7b76'\n", " exit_status='success'\n", " 2021-03-06 14:10:43.261 -- 2021-03-06 14:10:46.260\n", " Streams:\n", " * primary\n", " * baseline\n" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "run = cat[85]\n", "run" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Look at the data in the `primary` stream. That's the experiment data." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
<xarray.Dataset>\n",
       "Dimensions:           (time: 23)\n",
       "Coordinates:\n",
       "  * time              (time) float64 1.615e+09 1.615e+09 ... 1.615e+09 1.615e+09\n",
       "Data variables:\n",
       "    m1                (time) float64 0.915 0.918 0.922 ... 0.988 0.992 0.995\n",
       "    m1_user_setpoint  (time) float64 0.9146 0.9183 0.922 ... 0.988 0.9917 0.9954\n",
       "    noisy             (time) float64 1.464e+03 1.767e+03 ... 1.674e+03 1.494e+03
" ], "text/plain": [ "\n", "Dimensions: (time: 23)\n", "Coordinates:\n", " * time (time) float64 1.615e+09 1.615e+09 ... 1.615e+09 1.615e+09\n", "Data variables:\n", " m1 (time) float64 0.915 0.918 0.922 ... 0.988 0.992 0.995\n", " m1_user_setpoint (time) float64 0.9146 0.9183 0.922 ... 0.988 0.9917 0.9954\n", " noisy (time) float64 1.464e+03 1.767e+03 ... 1.674e+03 1.494e+03" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ds = run.primary.read()\n", "ds" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Plot it." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEGCAYAAABPdROvAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAYNElEQVR4nO3df5Dcd33f8ee7krDPEFkGhGudTGRiVcTGNMI3jsEMzcRpJIiDVRc6mg7BYZx64jr8SBsxVjITyj+1qRJ+dYpnXDsgA8E4jhAOhQhXpi1THJsTMgj/EFZQsHVSbFFbtkOvRpLf/WM/Z/bOp9PufXZvd++ej5md/e7n+/3uvXdvd1/7/Xy+3+9GZiJJ0mz9o14XIEkabAaJJKmKQSJJqmKQSJKqGCSSpCqLe13AXHvlK1+Zq1at6nUZkjRQdu3a9ePMXD7dvAUXJKtWrWJ0dLTXZUjSQImIH51onl1bkqQqBokkqYpBIkmqYpBIkqoYJJKkKgtury2p32zfPcaWHXs5eGScFcuG2LRuDRvWDve6LKllBonUQ9t3j7F52x7Gjx4HYOzIOJu37QEwTDQw7NqSemjLjr0vhMiE8aPH2bJjb48qktpnkEg9dPDIeFvtUj8ySKQeWrFsqK12qR8ZJFIPbVq3hqEliya1DS1ZxKZ1a3pUkdQ+B9ulHpoYUHevLQ0yg0TqsQ1rhw0ODTS7tiRJVQwSSVIVg0SSVMUgkSRVMUgkSVUMEklSFYNEklTFIJEkVTFIJElVDBJJUhWDRJJUxSCRJFUxSCRJVQwSSVIVg0SSVMUgkSRVMUgkSVUMEklSFYNEklTFIJEkVTFIJElVDBJJUpWuBklE/H5EPBAR34+IL0TEqRHx8oi4KyIeKddnNC2/OSL2RcTeiFjX1H5hROwp8z4ZEVHaT4mIL5b2eyNiVTcfjyTpxboWJBExDLwPGMnM1wGLgI3AdcDOzFwN7Cy3iYjzyvzzgfXApyJiUbm7G4GrgdXlsr60XwU8lZnnAh8DPtKtxyNJml63u7YWA0MRsRg4DTgIXA5sLfO3AhvK9OXAbZn5XGbuB/YBF0XEWcDSzLwnMxO4dco6E/d1B3DpxNaKJGludC1IMnMM+BPgUeAQ8HRmfh04MzMPlWUOAa8qqwwDjzXdxYHSNlymp7ZPWiczjwFPA6/oxuORJE2vm11bZ9DYYjgHWAG8NCLeNdMq07TlDO0zrTO1lqsjYjQiRg8fPjxz4ZKktnSza+vXgP2ZeTgzjwLbgDcBj5fuKsr1E2X5A8DZTeuvpNEVdqBMT22ftE7pPjsdeHJqIZl5U2aOZObI8uXLO/TwJEnQ3SB5FLg4Ik4r4xaXAg8BdwJXlmWuBL5cpu8ENpY9sc6hMah+X+n+ejYiLi738+4p60zc1zuAu8s4iiRpjizu1h1n5r0RcQfwHeAYsBu4CXgZcHtEXEUjbN5Zln8gIm4HHizLX5uZx8vdXQN8BhgCvlYuALcAn42IfTS2RDZ26/FIkqYXC+0L/MjISI6Ojva6DEkaKBGxKzNHppvnke2SpCoGiSSpikEiSapikEiSqhgkkqQqBokkqYpBIkmqYpBIkqoYJJKkKgaJJKmKQSJJqmKQSJKqGCSSpCoGiSSpikEiSapikEiSqhgkkqQqBokkqYpBIkmqYpBIkqoYJJKkKgaJJKmKQSJJqmKQSJKqGCSSpCoGiSSpikEiSapikEiSqhgkkqQqBokkqYpBIkmqYpBIkqoYJJKkKot7XYCkOtt3j7Flx14OHhlnxbIhNq1bw4a1w70uSwtIV7dIImJZRNwREQ9HxEMR8caIeHlE3BURj5TrM5qW3xwR+yJib0Ssa2q/MCL2lHmfjIgo7adExBdL+70Rsaqbj0fqN9t3j7F52x7GjoyTwNiRcTZv28P23WO9Lk0LSLe7tj4B/HVmvhb4p8BDwHXAzsxcDewst4mI84CNwPnAeuBTEbGo3M+NwNXA6nJZX9qvAp7KzHOBjwEf6fLjkfrKlh17GT96fFLb+NHjbNmxt0cVaSHqWpBExFLgLcAtAJn508w8AlwObC2LbQU2lOnLgdsy87nM3A/sAy6KiLOApZl5T2YmcOuUdSbu6w7g0omtFWkhOHhkvK12qRu6uUXyGuAw8OmI2B0RN0fES4EzM/MQQLl+VVl+GHisaf0DpW24TE9tn7ROZh4DngZeMbWQiLg6IkYjYvTw4cOdenxSz61YNtRWu9QN3QySxcAbgBszcy3wE0o31glMtyWRM7TPtM7khsybMnMkM0eWL18+c9XSANm0bg1DSxZNahtasohN69b0qCItRN0MkgPAgcy8t9y+g0awPF66qyjXTzQtf3bT+iuBg6V95TTtk9aJiMXA6cCTHX8kUp/asHaY66+4gOFlQwQwvGyI66+4wL22NKe6tvtvZv59RDwWEWsycy9wKfBguVwJ3FCuv1xWuRP484j4KLCCxqD6fZl5PCKejYiLgXuBdwP/uWmdK4F7gHcAd5dxFGnB2LB22OBQT3X7OJL3Ap+PiJcAPwTeQ2Mr6PaIuAp4FHgnQGY+EBG30wiaY8C1mTmxO8o1wGeAIeBr5QKNgfzPRsQ+GlsiG7v8eCRJU8RC+wI/MjKSo6OjvS5D84wHBWq+i4hdmTky3TyPbJcqTRwUOHE8x8RBgYBhogXBc21JlTwoUAudQSJV8qBALXQGiVTJgwK10BkkUiUPCtRC52C7VGliQN29trRQGSRSB3hQoBYyu7YkSVUMEklSlZaCpOkHpiRJmqTVLZJ9EbGl/IqhJEkvaDVIXg/8ALg5Iv6m/FDU0i7WJUkaEC0FSWY+m5n/NTPfBHwQ+BBwKCK2RsS5Xa1QktTXWh4jiYi3R8SXgE8Af0rjp3T/CvhqF+uTJPW5Vo8jeQT4BrAlM7/V1H5HRLyl82VJkgZFq0Hy+sz8h+lmZOb7OliPJGnAtDrY/scRsTQilkTEzoj4cUS8q6uVSZIGQqtB8uuZ+QxwGXAA+CfApq5VJUkaGK0GyZJy/TbgC5n5ZJfqkSQNmFbHSP4qIh4GxoF/GxHLgf/XvbIkSYOi1eNIrgPeCIxk5lHgJ8Dl3SxMkjQYZtwiiYhfzcy7I+KKprbmRbZ1qzBJ0mA4WdfWPwPuBn5zmnmJQSJJC96MQZKZHyrX75mbciRJg6bVU6ScHhEfjYjRcvnTiDi928VJkvpfq7v//hnwLPCvyuUZ4NPdKkqSNDha3f33FzLzXzbd/nBE3N+FeiRJA6bVLZLxiHjzxI2IuITGMSWSpAWu1S2Sa4CtTeMiTwFXdqckSdIgaTVIHgL+E/ALwDLgaWAD8L2uVCVJGhitBsmXgSPAd4CxrlUjSRo4rQbJysxc39VKJEkDqdXB9m9FxAVdrUSSNJBa3SJ5M/DbEbEfeA4IIDPz9V2rTJI0EFrdInkrsBr4dRrn3bqM6c+/9SIRsSgidkfEV8rtl0fEXRHxSLk+o2nZzRGxLyL2RsS6pvYLI2JPmffJKGeOjIhTIuKLpf3eiFjV4uORJHVIq6eR/9F0lxb/xvtp7PU14TpgZ2auBnaW20TEecBG4HxgPfCpiFhU1rkRuJpGmK0u8wGuAp7KzHOBjwEfabEmSVKHtLpFMisRsRL4DeDmpubLga1leiuN3Ygn2m/LzOcycz+wD7goIs4ClmbmPZmZwK1T1pm4rzuASye2ViRJc6OrQQJ8HPgg8HxT25mZeQigXL+qtA8DjzUtd6C0DZfpqe2T1snMYzSOb3nF1CIi4uqJE04ePny48iFJkpp1LUgi4jLgiczc1eoq07TlDO0zrTO5IfOmzBzJzJHly5e3WI4kqRWt7rU1G5cAb4+ItwGnAksj4nPA4xFxVmYeKt1WT5TlDwBnN62/EjhY2ldO0968zoGIWAycDjzZrQckSXqxrm2RZObmzFyZmatoDKLfnZnvAu7kZ+fpupLGUfOU9o1lT6xzaAyq31e6v56NiIvL+Me7p6wzcV/vKH/jRVskkqTu6eYWyYncANweEVcBjwLvBMjMByLiduBB4BhwbWYeL+tcA3wGGAK+Vi4AtwCfjYh9NLZENs7Vg5AkNcRC+wI/MjKSo6OjvS5DkgZKROzKzJHp5nV7ry1J0jxnkEiSqhgkkqQqBokkqYpBIkmq0ovdfyX1yPbdY2zZsZeDR8ZZsWyITevWsGHt8MlXlGZgkEgnMN8+dLfvHmPztj2MH20cnjV2ZJzN2/YADPTjUu/ZtSVNY+JDd+zIOMnPPnS37x7rdWmztmXH3hdCZML40eNs2bG3RxVpvjBIpGnMxw/dg0fG22qXWmWQSNOYjx+6K5YNtdUutcogkaYxHz90N61bw9CSRZPahpYsYtO6NT2qSPOFQSJNYz5+6G5YO8z1V1zA8LIhAhheNsT1V1zgQLuqudeWNI2JD9f5tNcWNB7XoD8G9R+DRDoBP3Sl1ti1JUmqYpBIkqoYJJKkKgaJJKmKQSJJqmKQSJKqGCSSpCoGiSSpikEiSapikEiSqhgkkqQqBokkqYpBIkmqYpBIkqoYJJKkKgaJJKmKQSJJqmKQSJKqGCSSpCpdC5KIODsivhERD0XEAxHx/tL+8oi4KyIeKddnNK2zOSL2RcTeiFjX1H5hROwp8z4ZEVHaT4mIL5b2eyNiVbcejyRpet3cIjkG/PvM/EXgYuDaiDgPuA7YmZmrgZ3lNmXeRuB8YD3wqYhYVO7rRuBqYHW5rC/tVwFPZea5wMeAj3Tx8UiSptG1IMnMQ5n5nTL9LPAQMAxcDmwti20FNpTpy4HbMvO5zNwP7AMuioizgKWZeU9mJnDrlHUm7usO4NKJrRVJ0tyYkzGS0uW0FrgXODMzD0EjbIBXlcWGgceaVjtQ2obL9NT2Setk5jHgaeAV0/z9qyNiNCJGDx8+3KFHJUmCOQiSiHgZ8JfABzLzmZkWnaYtZ2ifaZ3JDZk3ZeZIZo4sX778ZCVLktrQ1SCJiCU0QuTzmbmtND9euqso10+U9gPA2U2rrwQOlvaV07RPWiciFgOnA092/pFIkk6km3ttBXAL8FBmfrRp1p3AlWX6SuDLTe0by55Y59AYVL+vdH89GxEXl/t895R1Ju7rHcDdZRxFkjRHFnfxvi8BfgvYExH3l7Y/BG4Abo+Iq4BHgXcCZOYDEXE78CCNPb6uzczjZb1rgM8AQ8DXygUaQfXZiNhHY0tkYxcfjyRpGrHQvsCPjIzk6Ohor8uQpIESEbsyc2S6ed3cIpF6bvvuMbbs2MvBI+OsWDbEpnVr2LB2+OQr6kV8LnUiBonmre27x9i8bQ/jRxs9pGNHxtm8bQ+AH4Bt8rnUTDzXluatLTv2vvDBN2H86HG27Njbo4oGl8+lZmKQaN46eGS8rXadmM+lZmKQaN5asWyorXadmM+lZmKQaN7atG4NQ0sWTWobWrKITevW9KiiweVzqZk42K55a2IQ2D2N6vlcaiYeRyJJOqmZjiOxa0uSVMUgkSRVMUgkSVUMEklSFYNEklTF3X8lzQlP+jh/GSSSus6TPs5vBon6mt9i54eZTvro/3PwGSTqW36LnT886eP85mC7+panLp8/POnj/GaQqG/5LXb+8KSP85tdW5pzrY57rFg2xNg0oeG32MHjSR/nN4NEc6qdcY9N69ZMWhb8FjvINqwdbjk43MlisNi1pTnVzrjHhrXDXH/FBQwvGyKA4WVDXH/FBX6gzHMTXzbGjoyT/OzLxvbdY70uTSfgFonmVLvjHu18i9X84K7Cg8ctEs0p997RybiTxeBxi0SzMts+bMc9dDK1O1k4vjL33CJR22r6sB330MnU7Crs+EpvuEWittX2YTvuoZnU7Crs+EpvGCQL2Gy7AOzDVrfN9stG7WvTbrHZMUjmkXbeBDXnsfJAQfWrmtfmbN4TBk+DYyTzRLt9wzXnsfJ0F+pXNa/Ndt8TteMx23ePcckNd3POdf+NS264e6DHcdwi6YKabymzXbfdvuGaLgBPd6F+VfPabPc9UTMeU3tm637bEjJIWjBXXUY167b7JqjtnnLAXP1qtq/Ndt8TNV/G5jqEuh08dm2dxFx2GdWs2+6BfnZPSZO1+56oObi2WyE0nbnYJdogOYl2/2k1L5Caddt9E3g8hzRZu++Jmi9jcxlCc/G7PgPftRUR64FPAIuAmzPzhk7e/1x2GdWsO5u+YbunpMnaeU/UjMfUnOFhLrvgWjXQQRIRi4D/Avxz4ADw7Yi4MzMf7NTfaPefVvMCqT19iMEgza3ZvufmMoTmYnf9gQ4S4CJgX2b+ECAibgMuBzoWJO3+02peIO4NJS0ccxVCc3F+u8jMjt3ZXIuIdwDrM/N3yu3fAn45M39vynJXA1cDvPrVr77wRz/6UVt/p992tZOkdnTiMywidmXmyLTzBjxI3gmsmxIkF2Xme0+0zsjISI6Ojs5ViZI0L8wUJIO+19YB4Oym2yuBgz2qRZIWpEEPkm8DqyPinIh4CbARuLPHNUnSgjLQg+2ZeSwifg/YQWP33z/LzAd6XJYkLSgDHSQAmflV4Ku9rkOSFqpB79qSJPXYQO+1NRsRcRhob//f7ngl8ONeFzEN62qPdbWvX2uzrpn9fGYun27GgguSfhERoyfala6XrKs91tW+fq3NumbPri1JUhWDRJJUxSDpnZt6XcAJWFd7rKt9/Vqbdc2SYySSpCpukUiSqhgkkqQqBkkXRMT6iNgbEfsi4rpp5p8REV+KiO9FxH0R8brSfnZEfCMiHoqIByLi/X1S16nl9ndLXR/uh7qa5i+KiN0R8ZV+qSsi/i4i9kTE/RHR0dNNV9a1LCLuiIiHy+vsjb2uKyLWlOdp4vJMRHyg13WVeb9fXvPfj4gvRMSpfVLX+0tND3TyuZq1zPTSwQuNc379LfAa4CXAd4HzpiyzBfhQmX4tsLNMnwW8oUz/HPCDqev2qK4AXlamlwD3Ahf3uq6m+f8O+HPgK/3wfyy3/w54ZT+9vsrtrcDvlOmXAMv6oa4p9/P3NA5+6/XrfhjYDwyV27cDv90Hdb0O+D5wGo3TXP13YHWnX2vtXNwi6bwXfrUxM38KTPxqY7PzgJ0AmfkwsCoizszMQ5n5ndL+LPAQjRdzr+vKzPyHssyScunUXhqzrgsgIlYCvwHc3KF6OlJXF826rohYCrwFuKXM+2lmHul1XVOWuRT428zs1NknautaDAxFxGIaH9yd+pmKmrp+EfibzPy/mXkM+J/Av+hQXbNikHTeMPBY0+0DvDgMvgtcARARFwE/T+O3VF4QEauAtTS+/fe8rtJ9dD/wBHBXZvZFXcDHgQ8Cz3eonk7VlcDXI2JXNH6hsx/qeg1wGPh06Qq8OSJe2gd1NdsIfKFDNVXVlZljwJ8AjwKHgKcz8+u9rovG1shbIuIVEXEa8DYm/y7TnDNIOi+maZv67f0G4IzywfxeYDdw7IU7iHgZ8JfABzLzmX6oKzOPZ+Yv0XghXzR1nKIXdUXEZcATmbmrQ7V0pK4y75LMfAPwVuDaiHhLH9S1GHgDcGNmrgV+Aryob74HdTXuoPGbQm8H/qJDNVXVFRFn0NhKOAdYAbw0It7V67oy8yHgI8BdwF/TCJxj9NDAn0a+D530VxtLOLwHICKCRj/s/nJ7CY0Q+XxmbuuXupqWORIR/wNYT+ObUS/r2gi8PSLeBpwKLI2Iz2VmJ97sVc9XZh4s109ExJdodGX8rx7XdRpwoGlr8g46FySdeH29FfhOZj7eoZpq61oH7M/Mw2XeNuBNwOd6XBeZeQulizIi/mO5v97p5QDNfLzQCOcf0vgWMzGIdv6UZZYBLynT/wa4tUwHcCvw8T6razllUBYYAr4JXNbruqYs8yt0drC95vl6KfBzTdPfAtb3uq5y+5vAmjL9H4At/VBXabsNeE8fve5/GXiARgAHjR0V3tvrusrtV5XrVwMPA2d08nlr+/H08o/P1wuNPssf0Ngr449K2+8Cv1um3wg8Ul4A2yZeBMCbaWzefg+4v1ze1gd1vZ7GZvX3aGyF/HE/PF9T7uNX6GCQVD5frykfDN8tH0R/1A91lXm/BIyW/+X2Tn4AVdZ1GvB/gNM7+Vx1oK4Pl/bvA58FTumTur4JPFheY5d2+jlr9+IpUiRJVRxslyRVMUgkSVUMEklSFYNEklTFIJEkVTFIpD4TEa+NiHsi4rmI+INe1yOdjEe2S/3nSeB9wIYe1yG1xC0SaQ5FxKpo/BbIzeX3JD4fEb8WEf87Ih6JiIsy84nM/DZwtNf1Sq0wSKS5dy7wCRpnDHgt8K9pnNXgD4A/7GFd0qwYJNLc25+ZezLzeRqnUNmZjVNM7AFW9bQyaRYMEmnuPdc0/XzT7edx3FIDyCCRJFXx24/UZyLiH9M4Q+9S4PmI+ACN3/Pu1I+cSR3l2X8lSVXs2pIkVTFIJElVDBJJUhWDRJJUxSCRJFUxSCRJVQwSSVKV/w/awDJlcn8UagAAAABJRU5ErkJggg==", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "ds.plot.scatter(\"m1\", \"noisy\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Assuming #90 has the same type of scan, plot it." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAEGCAYAAABYV4NmAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAY1UlEQVR4nO3df5BdZ3nY8e8TWTgLqSz/kB17ZSKDFVGDaYR3XBNTmokTpACNVReI2qGojBhNXDeQpoiR0pmStsNgRyEE/sAzjg0IQmM8riK7BKJ4pM5kmjg2axYQtlGsYIO1EpaILduTWYwsP/3jvhffXVbru9p978/vZ+bOPfc55z33fX3k++z7nvecE5mJJEmL7ae6XQFJ0mAywUiSqjDBSJKqMMFIkqowwUiSqjij2xXoFeedd16uWrWq29WQpL7ywAMP/CAzV8y2zgRTrFq1ivHx8W5XQ5L6SkR891TrHCKTJFVRLcFExKcj4mhEfKsldk5E3BMRj5T3s1vWbY+IgxFxICLWtcSviIj9Zd0nIyJK/MyI+GKJ3xcRq1rKbCrf8UhEbKrVRknSqdXswXwWWD8jtg3Ym5mrgb3lMxFxGbAReG0p86mIWFLK3AxsAVaXV3Ofm4GnMvNS4OPATWVf5wAfBv45cCXw4dZEJknqjGoJJjP/CnhyRvhaYGdZ3glsaInfnpnPZeajwEHgyoi4EFiWmfdm4542n5tRprmvO4FrSu9mHXBPZj6ZmU8B9/CTiU6SVFmnz8FckJlHAMr7+SU+Cjzest2hEhstyzPj08pk5vPA08C5c+zrJ0TElogYj4jxY8eOLaBZkqSZemUWWcwSyznip1tmejDzFuAWgLGxMe/6qap2T0yyY88BDh+f4qLlI2xdt4YNa2f920caCJ3uwTxRhr0o70dL/BBwcct2K4HDJb5ylvi0MhFxBnAWjSG5U+1L6prdE5Ns37WfyeNTJDB5fIrtu/aze2Ky21WTqul0grkbaM7q2gTc1RLfWGaGXULjZP79ZRjt2Yi4qpxfec+MMs19vQPYV87T7AHeEhFnl5P7bykxqWt27DnA1ImT02JTJ06yY8+BLtVIqq/aEFlE/CnwS8B5EXGIxsyuG4E7ImIz8D3gnQCZ+WBE3AE8BDwP3JCZzf8br6cxI20E+Ep5AdwGfD4iDtLouWws+3oyIv4n8NWy3f/IzJmTDaSOOnx8al5xaRBUSzCZ+W9PseqaU2z/EeAjs8THgdfNEv8hJUHNsu7TwKfbrqxU2UXLR5icJZlctHykC7WROsMr+aUO2LpuDSNLl0yLjSxdwtZ1a7pUI6m+XplFJg205mwxZ5FpmJhgpA7ZsHbUhKKh4hCZJKkKezDSafCiSemlmWCkeWpeNNm8rqV50SRgkpFaOEQmzZMXTUrtsQcjzVMnL5p0KE79zB6MNE+nujhysS+a9P5l6ncmGGmeOnXRpENx6ncOkUnz1KmLJr1/mfqdCUY6DZ24aNL7l6nfOUQm9SjvX6Z+Zw9G6lHev0z9zgQj9TDvX6Z+5hCZJKkKE4wkqQoTjCSpChOMJKkKE4wkqQoTjCSpChOMJKkKE4wkqQoTjCSpChOMJKkKE4wkqQoTjCSpChOMJKkKE4wkqQoTjCSpCp8HIw2Y3ROTPqRMPcEEIw2Q3ROTbN+1n6kTJwGYPD7F9l37AUwy6jiHyKQBsmPPgR8nl6apEyfZsedAl2qkYWaCkQbI4eNT84pLNZlgpAFy0fKRecWlmkww0gDZum4NI0uXTIuNLF3C1nVrulQjDTNP8ksDpHki31lk6gVdSTAR8Z+B9wEJ7AfeC7wc+CKwCngMeFdmPlW23w5sBk4C78/MPSV+BfBZYAT4MvCBzMyIOBP4HHAF8A/Ab2TmY51pndRdG9aOmlDUEzo+RBYRo8D7gbHMfB2wBNgIbAP2ZuZqYG/5TERcVta/FlgPfCoimmMANwNbgNXltb7ENwNPZealwMeBmzrQNElSi26dgzkDGImIM2j0XA4D1wI7y/qdwIayfC1we2Y+l5mPAgeBKyPiQmBZZt6bmUmjx9JaprmvO4FrIiLqNkmS1KrjCSYzJ4E/AL4HHAGezsy/BC7IzCNlmyPA+aXIKPB4yy4OldhoWZ4Zn1YmM58HngbOnVmXiNgSEeMRMX7s2LHFaaAkCejOENnZNHoYlwAXAa+IiHfPVWSWWM4Rn6vM9EDmLZk5lpljK1asmLvikqR56cYQ2a8Aj2bmscw8AewCfhF4ogx7Ud6Plu0PARe3lF9JY0jtUFmeGZ9WpgzDnQU8WaU1kqRZdSPBfA+4KiJeXs6LXAM8DNwNbCrbbALuKst3Axsj4syIuITGyfz7yzDasxFxVdnPe2aUae7rHcC+cp5GktQhHZ+mnJn3RcSdwNeA54EJ4BbgZ4A7ImIzjST0zrL9gxFxB/BQ2f6GzGzebOl6Xpym/JXyArgN+HxEHKTRc9nYgaZJklqEf9g3jI2N5fj4eLerIUl9JSIeyMyx2dZ5qxhJUhUmGElSFSYYSVIVJhhJUhUmGElSFSYYSVIVJhhJUhUmGElSFT7RUgNn98SkT3SUeoAJRgNl98Qk23ftZ+pE425Ck8en2L5rP4BJRuowh8g0UHbsOfDj5NI0deIkO/Yc6FKNpOFlgtFAOXx8al5xSfWYYDRQLlo+Mq+4pHpMMBooW9etYWTpkmmxkaVL2LpuTZdqJA0vT/JroDRP5DuLrH3OulMtJhgNnA1rR/2BbJOz7lSTQ2TSEHPWnWoywUhDzFl3qskEIw0xZ92pJhOMNMScdaeaPMkvDTFn3akmE4w05Jx1p1ocIpMkVWGCkSRVYYKRJFVhgpEkVWGCkSRVYYKRJFVhgpEkVWGCkSRVYYKRJFVhgpEkVWGCkSRVYYKRJFVhgpEkVWGCkSRVYYKRJFXRlQQTEcsj4s6I+HZEPBwRb4yIcyLinoh4pLyf3bL99og4GBEHImJdS/yKiNhf1n0yIqLEz4yIL5b4fRGxqgvNlKSh1q0ezCeAv8jM1wD/DHgY2AbszczVwN7ymYi4DNgIvBZYD3wqIprPeL0Z2AKsLq/1Jb4ZeCozLwU+DtzUiUZJkl7U8QQTEcuANwO3AWTmjzLzOHAtsLNsthPYUJavBW7PzOcy81HgIHBlRFwILMvMezMzgc/NKNPc153ANc3ejSSpM9pKMC09hsXwKuAY8JmImIiIWyPiFcAFmXkEoLyfX7YfBR5vKX+oxEbL8sz4tDKZ+TzwNHDuzIpExJaIGI+I8WPHji1W+yRJtN+DORgRO8pw1UKdAbwBuDkz1wL/SBkOO4XZeh45R3yuMtMDmbdk5lhmjq1YsWLuWkuS5qXdBPN64O+AWyPib8tf/stO8zsPAYcy877y+U4aCeeJMuxFeT/asv3FLeVXAodLfOUs8WllIuIM4CzgydOsryTpNLSVYDLz2cz848z8ReBDwIeBIxGxMyIunc8XZub3gccjYk0JXQM8BNwNbCqxTcBdZfluYGOZGXYJjZP595dhtGcj4qpyfuU9M8o09/UOYF85TyNJ6pAz2tmonIN5G/BeYBXwMeALwL8Avgz8/Dy/97eAL0TEy4DvlP3+FHBHRGwGvge8EyAzH4yIO2gkoeeBGzLzZNnP9cBngRHgK+UFjQkEn4+IgzR6LhvnWT/1iN0Tk+zYc4DDx6e4aPkIW9etYcPa0ZcuKKnrop0/7CPiO8D/BW7LzL+Zse6Tmfn+SvXrmLGxsRwfH+92NdRi98Qk23ftZ+rEyR/HRpYu4aPXXW6SkXpERDyQmWOzrWv7HExmbp6ZXAAGIbmoN+3Yc2BacgGYOnGSHXsOdKlGkuaj3QTz3yJiWUQsjYi9EfGDiHh31Zpp6B0+PjWvuKTe0m6CeUtmPgO8ncYMrZ8HtlarlQRctHxkXnFJvaXdBLO0vL8V+NPMdMqvqtu6bg0jS6df4zuydAlb1605RQlJvaStWWTA/4mIbwNTwH+MiBXAD+tVS+LHJ/KdRSb1p7ZmkQGUuxs/k5knI+LlNO4D9v2qtesgZ5FJ0vzNNYtszh5MRPxyZu6LiOtaYq2b7FqcKkqSBs1LDZH9S2Af8K9mWZeYYCRJpzBngsnMD5f393amOpKkQdHu7frPiog/bN7aPiI+FhFn1a6cJKl/tTtN+dPAs8C7yusZ4DO1KiVJ6n/tTlN+dWb+m5bP/z0ivl6hPpKkAdFuD2YqIt7U/BARV9O4JkaSpFm124O5HtjZct7lKV583ookST+h3QTzMPD7wKuB5TSecb8B+GaVWkmS+l67CeYu4DjwNWCyWm0kSQOj3QSzMjPXV62JJGmgtHuS/28i4vKqNZEkDZR2ezBvAv5DRDwKPAcEkJn5+mo1kyT1tXYTzK9VrYUkaeC0lWAy87u1KyJpsO2emPTZPkOm3R6MtCj8kRlOuycm2b5rP1MnTgIweXyK7bv2A3j8B1i7J/mlBWv+yEwenyJ58Udm94Qz3wfdjj0HfpxcmqZOnGTHngNdqpE6wQSjjvFHZngdPj77naVOFddgMMGoY/yRGV4XLR+ZV1yDwQSjjvFHZnhtXbeGkaVLpsVGli5h67o1XaqROsEEo47xR2Z4bVg7ykevu5zR5SMEMLp8hI9ed7kn+Aecs8jUMc0fE2eRDacNa0c91kPGBKOO8kdGGh4OkUmSqjDBSJKqMMFIkqowwUiSqjDBSJKqMMFIkqowwUiSqjDBSJKq6FqCiYglETEREV8qn8+JiHsi4pHyfnbLttsj4mBEHIiIdS3xKyJif1n3yYiIEj8zIr5Y4vdFxKqON1CShlw3ezAfAB5u+bwN2JuZq4G95TMRcRmwEXgtsB74VEQ0b2h1M7AFWF1e60t8M/BUZl4KfBy4qW5TJEkzdSXBRMRK4G3ArS3ha4GdZXknsKElfntmPpeZjwIHgSsj4kJgWWbem5kJfG5Gmea+7gSuafZuJPWH3ROTXH3jPi7Z9udcfeM+H0zXh7rVg/kj4EPACy2xCzLzCEB5P7/ER4HHW7Y7VGKjZXlmfFqZzHweeBo4d2YlImJLRIxHxPixY8cW2CRJi8Wnnw6GjieYiHg7cDQzH2i3yCyxnCM+V5npgcxbMnMsM8dWrFjRZnUk1ebTTwdDN+6mfDXw6xHxVuCngWUR8SfAExFxYWYeKcNfR8v2h4CLW8qvBA6X+MpZ4q1lDkXEGcBZwJO1GiRpcfn008HQ8R5MZm7PzJWZuYrGyft9mflu4G5gU9lsE3BXWb4b2Fhmhl1C42T+/WUY7dmIuKqcX3nPjDLNfb2jfMdP9GC0MI6RqxaffjoYeuk6mBuBX42IR4BfLZ/JzAeBO4CHgL8AbsjMZt/5ehoTBQ4Cfw98pcRvA86NiIPA71BmpGnxOEaumnz66WAI/7BvGBsby/Hx8W5Xo29cfeM+JmcZrhhdPsJfb/vlLtRIg2b3xKRPP+0DEfFAZo7Nts4nWuq0OEau2nz6af/rpSEy9RHHyCW9FBOMTotj5JJeikNkOi3NoQvHyCWdiglGp80xcklzcYhMklSFPRhJA8Opzb3FBCNpIDQv/m3ew6x58S9gkukSh8gkDQRvkNl7TDCSBoIX//YeE4ykgeDFv73HBCNpIHjxb+/xJL8AZ9+o/3nxb+8xwcjZNxoYXvzbWxwik7NvJFVhgpGzbyRVYYKRs28kVWGCkbNvJFXhSX45+0ZSFSYYAc6+kbT4HCKTJFVhgpEkVWGCkSRV4TkYSUPN2yTVY4KRNLS8TVJdDpFJGlreJqkuE4ykoeVtkuoywUgaWt4mqS4TjKSh5W2S6vIkv6Sh5W2S6jLBSBpq871NktOa22eCkaQ2Oa15fjwHI0ltclrz/NiD6QN2yaXe4LTm+bEH0+OaXfLJ41MkL3bJd09Mdrtq0tBxWvP8mGB6nF1yqXc4rXl+HCLrcXbJpd7htOb56XiCiYiLgc8BPwu8ANySmZ+IiHOALwKrgMeAd2XmU6XMdmAzcBJ4f2buKfErgM8CI8CXgQ9kZkbEmeU7rgD+AfiNzHysQ01cVBctH2FylmRil1zqDp/+2r5uDJE9D/yXzPynwFXADRFxGbAN2JuZq4G95TNl3UbgtcB64FMR0eyj3gxsAVaX1/oS3ww8lZmXAh8HbupEw2qwSy6pX3U8wWTmkcz8Wll+FngYGAWuBXaWzXYCG8rytcDtmflcZj4KHASujIgLgWWZeW9mJo0eS2uZ5r7uBK6JiKjasEo2rB3lo9ddzujyEQIYXT7CR6+73L+gJPW8rp6DiYhVwFrgPuCCzDwCjSQUEeeXzUaBv20pdqjETpTlmfFmmcfLvp6PiKeBc4EfzPj+LTR6QLzyla9ctHYtNrvkkvpR1xJMRPwM8L+B387MZ+boYMy2IueIz1VmeiDzFuAWgLGxsZ9YX4PXtEgaFl2ZphwRS2kkly9k5q4SfqIMe1Hej5b4IeDiluIrgcMlvnKW+LQyEXEGcBbw5OK3ZH68pkXSMOl4ginnQm4DHs7MP2xZdTewqSxvAu5qiW+MiDMj4hIaJ/PvL8Npz0bEVWWf75lRprmvdwD7ynmarvKaFknDpBtDZFcD/x7YHxFfL7HfBW4E7oiIzcD3gHcCZOaDEXEH8BCNGWg3ZGbzV/p6Xpym/JXygkYC+3xEHKTRc9lYuU1t8ZoWSe0ahOH0jieYzPx/zH6OBOCaU5T5CPCRWeLjwOtmif+QkqB6ide0SGrHoNy12VvFdJDXtEhqx6AMp3urmAWaTzfW20xIasegDKebYBbgdLqxXtMi6aUMynC6Q2QLMCjdWEm95XSG03dPTHL1jfu4ZNufc/WN+3ri8gd7MAswKN1YSb1lvsPpvTopwASzAIPSjZXUe+YznD7XaEo3E4xDZAvgrDBJvaBXR1NMMAvgnY4l9YJefZSzQ2QL5KwwSd22dd2aaedgoDdGU0wwktTnTvcau9q3ozHBSNIAmO9oSidmnnkORpKGUCeu4zPBSNIQ6sTMMxOMJA2hTsw8M8FI0hDqxHV8nuSXpCHUibu7m2AkaUjVvo7PITJJUhUmGElSFSYYSVIVJhhJUhUmGElSFZGZ3a5DT4iIY8B3O/iV5wE/6OD39Zphbv8wtx2Gu/2D2Pafy8wVs60wwXRJRIxn5li369Etw9z+YW47DHf7h63tDpFJkqowwUiSqjDBdM8t3a5Alw1z+4e57TDc7R+qtnsORpJUhT0YSVIVJhhJUhUmmEUSEesj4kBEHIyIbbOsPzsi/iwivhkR90fE62asXxIRExHxpZbY70XEZER8vbze2om2zNdC2h4Rj0XE/tK+8Zb4ORFxT0Q8Ut7P7lR75qNS2/viuMOC2788Iu6MiG9HxMMR8cYSH4Zjf6q2982xb0tm+lrgC1gC/D3wKuBlwDeAy2ZsswP4cFl+DbB3xvrfAf4X8KWW2O8BH+x2+2q2HXgMOG+W/f4+sK0sbwNu6nZbO9j2nj/ui9T+ncD7yvLLgOVDdOxP1fa+OPbtvuzBLI4rgYOZ+Z3M/BFwO3DtjG0uA/YCZOa3gVURcQFARKwE3gbc2rkqL5oFtX0O19L4n5DyvmHRarx4arW9X5x2+yNiGfBm4Lay7keZebyUGehj/xJtHygmmMUxCjze8vlQibX6BnAdQERcCfwcsLKs+yPgQ8ALs+z7P5Uu9qd7dKhgoW1P4C8j4oGI2NJS5oLMPAJQ3s+vUPeFqtV26P3jDgtr/6uAY8BnytDwrRHxilJm0I/9XG2H/jj2bTHBLI6YJTZz/veNwNkR8XXgt4AJ4PmIeDtwNDMfmGUfNwOvBn4BOAJ8bLEqvIhOu+1l3dWZ+Qbg14AbIuLNtSpaQa2298Nxh4W1/wzgDcDNmbkW+Ecaw2H9olbb++XYt8VHJi+OQ8DFLZ9XAodbN8jMZ4D3AkREAI+W10bg18vJvJ8GlkXEn2TmuzPziWb5iPhj4Ev0noW0ncw8XN6PRsSf0Rh6+CvgiYi4MDOPRMSFwNHaDTkNVdreJ8cdFtb+lwOHMvO+sumdvPgjO+jH/pRt76Nj3xZ7MIvjq8DqiLgkIl5GI2nc3bpBmTXysvLxfTR+SJ7JzO2ZuTIzV5Vy+zLz3aXMhS27+NfAt2o35DScdtsj4hUR8U/KNq8A3sKLbbwb2FSWNwF3VW7H6ajS9j457rCwf/ffBx6PiDVl3TXAQ2V5oI/9XG3vo2Pfnm7PMhiUF/BW4O9ozCz5ryX2m8BvluU3Ao8A3wZ2AWfPso9fYvosss8D+4Fv0vjHe2G327mYbacxFv2N8nqwWbasO5fGCdJHyvs53W5nB9veF8d9Ie0v634BGC/t3N3y32agj/1LtL1vjn07L28VI0mqwiEySVIVJhhJUhUmGElSFSYYSVIVJhhJUhUmGKmPRMRrIuLeiHguIj7Y7fpIc/FKfqm/PAm8n968AaQ0jT0YqUdExKryfJBbI+JbEfGFiPiViPjraDwb5crMPJqZXwVOdLu+0ksxwUi95VLgE8DraTxD5N8BbwI+CPxuF+slzZsJRuotj2bm/sx8gcYtZPZm43Yb+4FVXa2ZNE8mGKm3PNey/ELL5xfwnKn6jAlGklSFfxFJfSQifpbGXXiXAS9ExG/TeBb8M12tmDQL76YsSarCITJJUhUmGElSFSYYSVIVJhhJUhUmGElSFSYYSVIVJhhJUhX/H7izB8s8pS2wAAAAAElFTkSuQmCC", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "cat[90].primary.read().plot.scatter(\"m1\", \"noisy\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Export one run to CSV\n", "\n", "Make a function to do this, then call it with the `run`. The function should export all data streams and the run's metadata. Each to a separate CSV file.\n", "\n", "1. Starting with a catalog instance, `cat`, pick a `run` by `scan_id` (or `uid`).\n", "2. Look at the stream names, the run's experimental data is in the `primary` stream.\n", "3. The stream data is obtained in an xarray Dataset: `ds = run.primary.read()`\n", "4. The stream data can be converted to CSV (by first converting to a pandas DataFrame): `ds.to_pandas().to_csv()`\n", "5. Write the CSV to a file.\n", "6. The run's metadata is already a dictionary (making it difficult to render as CSV). The `csv` package has a special writer for dictionaries." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "wrote file: run-85-metadata.csv\n", "wrote file: run-85-primary.csv\n", "wrote file: run-85-baseline.csv\n" ] } ], "source": [ "import csv\n", "from xarray.core.dataset import Dataset as xarray_Dataset\n", "from pandas import DataFrame as pandas_DataFrame\n", "\n", "def export_run(run):\n", " \"Save all the run information to text files.\"\n", " scan_id = run.metadata[\"start\"][\"scan_id\"]\n", "\n", " def export_data(data, scan_id, label, prefix=\"run\"):\n", " with open(f\"{prefix}-{scan_id}-{label}.csv\", \"w\", newline=\"\\n\") as f:\n", " if isinstance(data, dict):\n", " writer = csv.DictWriter(f, data.keys())\n", " writer.writeheader()\n", " writer.writerow(data)\n", " elif isinstance(data, xarray_Dataset):\n", " f.write(data.to_pandas().to_csv())\n", " elif isinstance(data, pandas_DataFrame):\n", " f.write(data.to_csv())\n", " print(f\"wrote file: {f.name}\")\n", "\n", " export_data(run.metadata, scan_id, \"metadata\")\n", " for stream_name in run:\n", " ds = getattr(run, stream_name).read()\n", " if stream_name == \"baseline\":\n", " ds = ds.to_pandas().transpose()\n", " export_data(ds, scan_id, stream_name)\n", "\n", "export_run(run)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Search for runs by time\n", "\n", "Since a search by scan_id` might not be unique, we hope to resolve that by restricting the time span for a search. Here's how to search by a time span." ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(59, 8)" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "time_span = databroker.queries.TimeRange(since=\"2021-03-06 14:00\", until=\"2021-03-06 16:00\")\n", "runs = cat.search(time_span)\n", "len(cat), len(runs)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Export a list of runs\n", "\n", "Search the catalog by time and `scan_id` for a **_list_** of runs. Call the `export_run()` function on each run. We'll look for #70 - #84 using the `range(70, 86)` function. But, since we have a time restriction, only some of the scans will be written.\n", "\n", "The result of `cat.search(query)` (where `query` is a dictionary of search terms) is another catalog with runs matching the search terms. This result can be searched again and again as needed. The dictionary can use [mongoquery](https://docs.mongodb.com/manual/reference/operator/query/) as shown here: `{'scan_id': {'$in': scan_id_list}}`\n", "\n", "Here, a time span is selected along with a list of scan IDs. The time span is searched first, since any `scan_id` might not be unique in the catalog. (_assumption_: A `scan_id` might be unique within the time span.) The result is a catalog of scans from the list but also within the time span." ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "wrote file: run-83-metadata.csv\n", "wrote file: run-83-primary.csv\n", "wrote file: run-83-baseline.csv\n", "wrote file: run-84-metadata.csv\n", "wrote file: run-84-primary.csv\n", "wrote file: run-84-baseline.csv\n", "wrote file: run-85-metadata.csv\n", "wrote file: run-85-primary.csv\n", "wrote file: run-85-baseline.csv\n" ] } ], "source": [ "def export_runs(scan_id_list, since=None, until=None):\n", " runs = cat\n", " # first by date since scan_id might not be unique for all times\n", " if since is not None:\n", " runs = runs.search(databroker.queries.TimeRange(since=since))\n", " if until is not None:\n", " runs = runs.search(databroker.queries.TimeRange(until=until))\n", " # now, search by scan_id (in the supplied list)\n", " runs = runs.search({'scan_id': {'$in': scan_id_list}})\n", "\n", " for uid in runs:\n", " export_run(runs[uid])\n", "\n", "export_runs(range(70, 86), since=\"2021-03-06 14:00\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Export run to JSON file\n", "\n", "A Python dictionary can be written to JSON (its contents can be expressed in JSON. There are a few notable problems, such as `None` values.) Here, a dictionary (`data`) is built with the different parts of a run (metadata, streams). Then `data` is written to a single file." ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "wrote file run_85.json\n" ] } ], "source": [ "import json\n", "\n", "def export_json(run):\n", " \"\"\"Export all data from a run to a JSON file.\"\"\"\n", " scan_id = run.metadata[\"start\"][\"scan_id\"]\n", " data = {\"metadata\": run.metadata}\n", " for stream_name in list(run): # get ALL the streams\n", " # such as data[\"primary\"] = run.primary.read().to_dict()\n", " data[stream_name] = getattr(run, stream_name).read().to_dict()\n", " with open(f\"run_{scan_id}.json\", \"w\", newline=\"\\n\") as f:\n", " f.write(json.dumps(data, indent=2))\n", " print(f\"wrote file {f.name}\")\n", "\n", "export_json(run)\n" ] } ], "metadata": { "interpreter": { "hash": "fd0bb865bd950a0eb522eff7964c260f306407a171414059e5c2e75397d9f14a" }, "kernelspec": { "display_name": "Python 3.9.7 64-bit ('bluesky_2022_1': conda)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.7" }, "orig_nbformat": 4 }, "nbformat": 4, "nbformat_minor": 2 }