I have written the fundamentals of a dashboard to crawl some sites and collect information, saved to a database.
The project is hosted on GCP and has a CICD process already established.
I have used Django for the ORM with Postgres and Playwrite for aggregating the data. Since there are long running processes Celery will need to be used with RabbitMQ for the backend and Redis for the results server.
You will need to have a good understanding of Django, Celery and Playwrite (or Puppeteer as they're very similar). The GCP infrastructure is already set up with the repo currently hosted in GCPs Source Repository.
The goal is to get the dashboard functioning; this means aggregating two sites and exposing the data in some Django Templates with the ability to add some extra data via fields.
The project will be broken down into milestones so we both understand the requirements of each stage.
I have reviewed project description and have some questions which i would like to discuss further. Please message me so we can discuss further. Thank you.