HyFI: Hydra Fast Interface

HyFI, short for Hydra Fast Interface, is a powerful Python framework built atop the foundations of Hydra and Pydantic. Its main objective is to provide a streamlined interface for configuring, structuring, executing, and scaling Python applications and workflows.

Overview

In the modern world of software and data science, creating scalable, reproducible, and modular Python workflows is vital. HyFI is here to make that process not just feasible but also efficient and straightforward. It has been meticulously crafted to help developers and researchers build pipeline-oriented projects and promote shareable and reproducible workflows.

Core Features

1. Dynamic Configuration Management

Modular Configuration: Harness the power of Hydra and Pydantic to manage your application configurations dynamically and modularity.
Seamless Command Line Overrides: Experiment swiftly without the clutter of multiple similar configuration files.
Jupyter Notebook Integration: The HyFI class allows for easy composition of configurations even in a jupyter notebook setting.

2. Zero Boilerplate

Stay focused on your core problems. HyFI takes care of the usual overheads such as command line flags, loading configuration files, and logging.

3. Structured Workflows

Configurable Automated Processes: Divide your research into unit jobs or tasks, and then bundle them into workflows.
Versatility: Have the liberty to create multiple workflows, with each performing varied sets of tasks.

Sharing Capability: Share your datasets, models, and configurations effortlessly.
Reproducibility: Sharing configurations alongside datasets and models ensures that every piece of research can be replicated.
Granular or Holistic: Choose to share individual unit jobs or an entire workflow, as per your need.

5. Support for Plugins

Integration of HyFI-based Applications: Enhance your project by plugging in several HyFI-based applications. This allows you to leverage the functionalities of plugins along with their specific configuration files.
Modular Design Enablement: The plugin support fosters a truly modular design approach, enabling seamless integration of various components and features, thereby promoting greater flexibility and extensibility in your projects.

Other Noteworthy Features:

Workspace Management: Automatic workspace creation and environment variable management.
Task Pipelining: Chain and parallelize jobs and steps seamlessly.
Optimized Parallelism: Efficient job batching with the help of Joblib.
Dotenv Integration: Manage configurations via .env files with ease.
Built-in Commands: Simplify processes like initializing projects, running pipelines, and copying files.
Extensibility: Easily extend HyFI's capabilities by crafting new Config classes and utilizing the building blocks it offers.
Command Line Interface (CLI): Access a suite of commands and workflows directly from the command line.
Advanced Logging: Make use of the integrated logging configurations, complemented by Hydra logging.
Robust Caching: Cache both files and data efficiently.

Citation

@software{lee_2023_8247719,
  author       = {Young Joon Lee},
  title        = {HyFI: Hydra Fast Interface},
  month        = aug,
  year         = 2023,
  publisher    = {Zenodo},
  doi          = {10.5281/zenodo.8247719},
  url          = {https://doi.org/10.5281/zenodo.8247719}
}

@software{lee_2023_hyfi,
  author       = {Young Joon Lee},
  title        = {HyFI: Hydra Fast Interface},
  year         = 2023,
  publisher    = {GitHub},
  url          = {https://github.com/entelecheia/hyfi}
}

Changelog

See the CHANGELOG for more information.

Contributing

Contributions are welcome! Please see the contributing guidelines for more information.

License

This project is released under the MIT License.

Check out the usage section for further information.

Note

This project is under active development.