Description:
We are seeking a skilled freelancer to turn our MVP code into a scalable system that tracks YouTube creators for a comprehensive keyword database. The goal is to create an automated, reliable solution that scans YouTube monthly for the top 100,000 creators associated with each keyword in our database.
Project Requirements:
1) Scalable System Design:
Transform existing MVP code into a system that searches each keyword from our “All cooking keywords” Google Sheet and identifies the top 10 creators per keyword.
Ensure the system can handle a significant increase in keywords without overloading the YouTube API quota.
Database Setup & Storage in BigQuery:
2) Set up Google BigQuery to store and organize the resulting creator data monthly.
Ensure BigQuery tables are organized by keyword, creator details, engagement metrics, etc., for efficient querying and analysis.
Provide guidance on monthly costs associated with storage and querying based on 100,000 entries.
API Management:
3) Advise on the optimal number of YouTube API keys needed to handle monthly data loads within quota limits.
Implement efficient API key rotation and error handling.
Cloud Automation & Scheduling:
Automate the code to run monthly using Google Cloud Functions or Cloud Run.
Integrate logging and monitoring for system health, API usage, and any quota issues.
Output Requirements:
Output to a Google BigQuery table for analysis, with fields such as channel ID, title, subscribers, views, country, and keyword.
Maintain user-friendly table organization for easy analysis and trend monitoring.
Cost Estimation:
Provide a detailed estimation of Google BigQuery costs, including storage and query costs for a high volume of data.
Skills Required:
Experience with YouTube Data API and API quota management.
Proficiency with Google Cloud Platform, BigQuery, Cloud Functions, and Cloud Run.
Python coding and experience with Google Sheets API and OAuth for authentication.
Strong understanding of scalable data processing and storage solutions.
Deliverables:
Fully automated, scalable code in Python.
Structured BigQuery tables for efficient querying.
Documentation on system operation, cost estimates, and error handling.
This is a long-term project with potential monthly work if the system operates well. Please provide relevant experience, estimated timeline, and a brief description of your approach to this project.
