Distiller is designed to streamline data acquisition and processing workflows for microscopy data. It integrates real-time monitoring, remote job launching, and post-processing capabilities, leveraging NERSC's high-performance computing (HPC) resources.
- Real-time Monitoring: Detects new microscopy data and displays metadata.
- Automated HPC Processing: Launches jobs at NERSC for data transfer and electron counting using stempy.
- Streaming Sessions: Enables real-time data processing on NERSC compute nodes.
- Jupyter Notebooks: Facilitates post-processing via NERSC's Jupyter hub.
- Frontend: React
- Backend: FastAPI, Kafka, Microservices
- Integration: Superfacility API (NERSC)
- Scans Page
- Shows recent acquisitions. One can filter acquisitions to a date range on this page.
Distiller.Scans.mp4
- Scan Page and Jupyter@NERSC
- Shows individual scan, and launching an electron counting job on NERSC compute nodes. After the data is processed, we can create a templated Jupyter notebook running on NERSC's Jupyterhub for post-processing
Distiller.Scan.and.Jupyter.mp4
- Streaming Session
- A streaming session runs in Distiller (left) while collecting data on the TEAM 0.5 microscope with DigitalMicrograph (right).