Artificial Intelligence playground for Data Scientist
Project here: https://confluence.suse.com/display/AAI/HackWeek19 Will keep working out of HackWeek as "best effort" personal project to make it evolve and keep learning.
What this project is about?
Data Scientist ofter starts working on their laptop before moving into company resources. As in many other cases they have to solve many challenges by themselves before actually start working on "their stuff". The idea is to build a prototype we will eventually try to evolve in a product that answers the following pre-requisites:
- Rapid Time to work: I, as Data Scientist or Data Engineer, need to install the playground quickly and be ready to work
- Everything at the right place: I as Data Scientist or Data Engineer want an easy way to find things and use them
- No time to waste: I as Data Scientist or Data Engineer want to be able to replicate the model synchronizing it with another infrastructure through a "click and done" model
- No complexity rule: I as Data Scientist or Data Engineer want to avoid waste time in complex configurations or debug things. Complexity needs to hided to me
Project Team requirements
Because this is a first attempt to prototype I have to ask for some "not official" rules to be applied:
- Max 7/9 people in the team with a max of 3 Engineers
- If you apply you have to make yourself available from 10 am to 5 pm CET (if you're on a different time zone you have to consider we'll have a lot of team discussion so could be challenging)
- This is a 5 days sprint approach where everyone needs to be open, collaborative, bold, creative.
FAQ
- I'm not an engineer or an expert: Great this project require (possibly) at least 1 person from marketing, sales-engineering, services, support
- Am I required to code?: No, but you're required to share your ideas and views, while the end goal is to build a prototype (that's why we need a couple of engineers) the scope is to have something to show and demonstrate we may build something useful for the Data Scientist community
- Woah this seems to be a super serious project: Nah it's a fun experiment to learn how much we may push our limit through rapid prototyping and "be different"
- So how do I signup?: easy just join the team here on hackweek and/or contact me alessandro.festa@suse.com for further details.
Looking for hackers with the skills:
ai artificial-intelligence machinelearning prototype agile projectmanagement innovation
This project is part of:
Hack Week 19
Activity
Comments
-
almost 5 years ago by hennevogel | Reply
Can you explain what kind of output you would expect? Like an application? A set of packages? Some IaC description?
-
almost 5 years ago by afesta | Reply
This is something we have to decide during the hack week, usually a prototype based on a target of the challenge decided by the team. If this will be simple artifacts made of a sum of existing items, an application or a set of packages has to be decided. The scope is to foster innovation under a very fast cycle (5 days) and get a result that allows us to learn if: is doable, what we need to address to make it a real product and how long could take. Don't expect huge development or impossible challenges, this is about pure innovation and ideas.. and build a way to demonstrate our idea.
-
-
almost 5 years ago by afesta | Reply
This is something we have to decide during the hack week, usually a prototype based on a target of the challenge decided by the team. If this will be simple artifacts made of a sum of existing items, an application or a set of packages has to be decided. The scope is to foster innovation under a very fast cycle (5 days) and get a result that allows us to learn if: is doable, what we need to address to make it a real product and how long could take. Don't expect huge development or impossible challenges, this is about pure innovation and ideas.. and build a way to demonstrate our idea.
-
almost 5 years ago by bmwiedemann | Reply
If you have a need for this project for 2x NVIDIA Tesla T4, 16GB - ping me.
-
-
almost 5 years ago by rsblendido | Reply
Is this about Kubeflow?
-
almost 5 years ago by afesta | Reply
Could be. I mean the only "constraint" is that ideally should work on a laptop and Kubeflow works on K8's but if you use something like MLRun you may overcome many challenges. The ultimate goal of the project is to provide Data scientists a playground so that they do not need to learn and install and configure everything but it's easy enough to start from your laptop (and eventually) move it to a server/cloud environment.
-
-
Similar Projects
ghostwrAIter - a local AI assisted tool for helping with support cases by paolodepa
Description
This project is meant to fight the loneliness of the support team members, providing them an AI assistant (hopefully) capable of scraping supportconfigs in a RAG fashion, trying to answer specific questions.
Goals
- Setup an Ollama backend, spinning one (or more??) code-focused LLMs selected by license, performance and quality of the results between:
- deepseek-coder-v2
- dolphin-mistral
- starcoder2
- (...others??)
- Setup a Web UI for it, choosing an easily extensible and customizable option between:
- Extend the solution in order to be able to:
- Add ZIU/Concord shared folders to its RAG context
- Add BZ cases, splitted in comments to its RAG context
- A plus would be to login using the IDP portal to ghostwrAIter itself and use the same credentials to query BZ
- Add specific packages picking them from IBS repos
- A plus would be to login using the IDP portal to ghostwrAIter itself and use the same credentials to query IBS
- A plus would be to desume the packages of interest and the right channel and version to be picked from the added BZ cases
AI for product management by a_jaeger
Description
Learn about AI and how it can help myself
What are the jobs that a PM does where AI can help - and how?
Goals
- Investigate how AI can help with different tasks
- Check out different AI tools, which one is best for which job
- Summarize learning
Resources
- Reading some blog posts by PMs that looked into it
- Popular and less popular AI tools
SUSE AI Meets the Game Board by moio
Use tabletopgames.ai’s open source TAG and PyTAG frameworks to apply Statistical Forward Planning and Deep Reinforcement Learning to two board games of our own design. On an all-green, all-open source, all-AWS stack!
AI + Board Games
Board games have long been fertile ground for AI innovation, pushing the boundaries of capabilities such as strategy, adaptability, and real-time decision-making - from Deep Blue's chess mastery to AlphaZero’s domination of Go. Games aren’t just fun: they’re complex, dynamic problems that often mirror real-world challenges, making them interesting from an engineering perspective.
As avid board gamers, aspiring board game designers, and engineers with careers in open source infrastructure, we’re excited to dive into the latest AI techniques first-hand.
Our goal is to develop an all-open-source, all-green AWS-based stack powered by some serious hardware to drive our board game experiments forward!
Project Goals
Set Up the Stack:
- Install and configure the TAG and PyTAG frameworks on SUSE Linux Enterprise Base Container Images.
- Integrate with the SUSE AI stack for GPU-accelerated training on AWS.
- Validate a sample GPU-accelerated PyTAG workload on SUSE AI.
- Ensure the setup is entirely repeatable with Terraform and configuration scripts, documenting results along the way.
Design and Implement AI Agents:
- Develop AI agents for the two board games, incorporating Statistical Forward Planning and Deep Reinforcement Learning techniques.
- Fine-tune model parameters to optimize game-playing performance.
- Document the advantages and limitations of each technique.
Test, Analyze, and Refine:
- Conduct AI vs. AI and AI vs. human matches to evaluate agent strategies and performance.
- Record insights, document learning outcomes, and refine models based on real-world gameplay.
Technical Stack
- Frameworks: TAG and PyTAG for AI agent development
- Platform: SUSE AI
- Tools: AWS for high-performance GPU acceleration
Why This Project Matters
This project not only deepens our understanding of AI techniques by doing but also showcases the power and flexibility of SUSE’s open-source infrastructure for supporting high-level AI projects. By building on an all-open-source stack, we aim to create a pathway for other developers and AI enthusiasts to explore, experiment, and deploy their own innovative projects within the open-source space.
Our Motivation
We believe hands-on experimentation is the best teacher.
Combining our engineering backgrounds with our passion for board games, we’ll explore AI in a way that’s both challenging and creatively rewarding. Our ultimate goal? To hack an AI agent that’s as strategic and adaptable as a real human opponent (if not better!) — and to leverage it to design even better games... for humans to play!
Research how LLMs could help to Linux developers and/or users by anicka
Description
Large language models like ChatGPT have demonstrated remarkable capabilities across a variety of applications. However, their potential for enhancing the Linux development and user ecosystem remains largely unexplored. This project seeks to bridge that gap by researching practical applications of LLMs to improve workflows in areas such as backporting, packaging, log analysis, system migration, and more. By identifying patterns that LLMs can leverage, we aim to uncover new efficiencies and automation strategies that can benefit developers, maintainers, and end users alike.
Goals
- Evaluate Existing LLM Capabilities: Research and document the current state of LLM usage in open-source and Linux development projects, noting successes and limitations.
- Prototype Tools and Scripts: Develop proof-of-concept scripts or tools that leverage LLMs to perform specific tasks like automated log analysis, assisting with backporting patches, or generating packaging metadata.
- Assess Performance and Reliability: Test the tools' effectiveness on real-world Linux data and analyze their accuracy, speed, and reliability.
- Identify Best Use Cases: Pinpoint which tasks are most suitable for LLM support, distinguishing between high-impact and impractical applications.
- Document Findings and Recommendations: Summarize results with clear documentation and suggest next steps for potential integration or further development.
Resources
- Local LLM Implementations: Access to locally hosted LLMs such as LLaMA, GPT-J, or similar open-source models that can be run and fine-tuned on local hardware.
- Computing Resources: Workstations or servers capable of running LLMs locally, equipped with sufficient GPU power for training and inference.
- Sample Data: Logs, source code, patches, and packaging data from openSUSE or SUSE repositories for model training and testing.
- Public LLMs for Benchmarking: Access to APIs from platforms like OpenAI or Hugging Face for comparative testing and performance assessment.
- Existing NLP Tools: Libraries such as spaCy, Hugging Face Transformers, and PyTorch for building and interacting with local LLMs.
- Technical Documentation: Tutorials and resources focused on setting up and optimizing local LLMs for tasks relevant to Linux development.
- Collaboration: Engagement with community experts and teams experienced in AI and Linux for feedback and joint exploration.
Learn how to integrate Elixir and Phoenix Liveview with LLMs by ninopaparo
Description
Learn how to integrate Elixir and Phoenix Liveview with LLMs by building an application that can provide answers to user queries based on a corpus of custom-trained data.
Goals
Develop an Elixir application via the Phoenix framework that:
- Employs Retrieval Augmented Generation (RAG) techniques
- Supports the integration and utilization of various Large Language Models (LLMs).
- Is designed with extensibility and adaptability in mind to accommodate future enhancements and modifications.
Resources
- https://elixir-lang.org/
- https://www.phoenixframework.org/
- https://github.com/elixir-nx/bumblebee
- https://ollama.com/
FamilyTrip Planner: A Personalized Travel Planning Platform for Families by pherranz
Description
FamilyTrip Planner is an innovative travel planning application designed to optimize travel experiences for families with children. By integrating APIs for flights, accommodations, and local activities, the app generates complete itineraries tailored to each family’s unique interests and needs. Recommendations are based on customizable parameters such as destination, trip duration, children’s ages, and personal preferences. FamilyTrip Planner not only simplifies the travel planning process but also offers a comprehensive, personalized experience for families.
Goals
This project aims to: - Create a user-friendly platform that assists families in planning complete trips, from flight and accommodation options to recommended family-friendly activities. - Provide intelligent, personalized travel itineraries using artificial intelligence to enhance travel enjoyment and minimize time and cost. - Serve as an educational project for exploring Go programming and artificial intelligence, with the goal of building proficiency in both.
Resources
To develop FamilyTrip Planner, the project will leverage: - APIs such as Skyscanner, Google Places, and TripAdvisor to source real-time information on flights, accommodations, and activities. - Go programming language to manage data integration, API connections, and backend development. - Basic machine learning libraries to implement AI-driven itinerary suggestions tailored to family needs and preferences.
Explore simple and distro indipendent declarative Linux starting on Tumbleweed or Arch Linux by janvhs
Description
Inspired by mkosi the idea is to experiment with a declarative approach of defining Linux systems. A lot of tools already make it possible to manage the systems infrastructure by using description files, rather than manual invocation. An example for this are systemd presets for managing enabled services or the /etc/fstab file for describing how partitions should be mounted.
If we would take inspiration from openSUSE MicroOS and their handling of the /etc/ directory, we could theoretically use systemd-sysupdate to swap out the /usr/ partition and create an A/B boot scheme, where the /usr/ partition is always freshly built according to a central system description. In the best case it would be possible to still utilise snapshots, but an A/B root scheme would be sufficient for the beginning. This way you could get the benefit of NixOS's declarative system definition, but still use the distros package repositories and don't have to deal with the overhead of Flakes or the Nix language.
Goals
- A simple and understandable system
- Check fitness of
mkosior write a simple extensible image builder tool for it - Create a declarative system specification
- Create a system with swappable
/usr/partition - Create an A/B root scheme
- Swap to the new system without reboot (kexec?)
Resources
- Ideas that have been floating around in my head for a while
- https://0pointer.net/blog/fitting-everything-together.html
- GNOME OS
- MicroOS
- systemd mkosi
- Vanilla OS