Context
In India, a new process similar to KYC (Know Your Customer) is being introduced called KYV (Know Your Vehicle). This process involves the validation of various vehicle-related documents such as the Registration Certificate (RC), axle count, number plates, type of number plates, and barcodes on FASTags. The bank project aimed to enable the processing of around 14,000 FASTags daily, along with a backlog of 75 lakh FASTags that need to be processed. Given the scale of this task, it was crucial to develop an architecture capable of handling such a massive load efficiently. My role was to design and deploy the AWS infrastructure for this system, ensuring that it could manage the networking (VPCs, subnets, etc.), load balancing, scaling, security, and deployment aspects required for the successful processing of FASTags. I also guided the team in building and implementing this infrastructure.
Requirements
High-Volume FASTag Processing:
Develop an infrastructure capable of processing 14,000 FASTags daily, along with validating associated documents such as vehicle RCs, axle count, number plates, and barcodes.
Handle a backlog of 75 lakh FASTags that need to be processed efficiently.
Scalable Architecture:
Design an architecture that can scale up and down based on the load, ensuring efficient resource utilization.
Implement load balancing to manage traffic and ensure system reliability under high loads.
Security and Compliance:
Ensure the architecture adheres to stringent security standards, protecting sensitive vehicle and customer data.
Implement security best practices, including VPCs, IAM policies, encryption, and secure API gateways.
Robust Deployment and Management:
Design a robust deployment pipeline using tools like Docker, Jenkins, and Ansible to ensure smooth and reliable deployment of the infrastructure.
Guide the team in setting up and managing the infrastructure, ensuring adherence to best practices.
Approach
Requirement Analysis:
Conducted thorough discussions with stakeholders to understand the specific needs of the KYV process and the volume of FASTags to be processed.
Identified key infrastructure requirements, including scalability, security, and deployment needs.
Design and Architecture:
Designed a scalable AWS-based architecture, utilizing VPCs, subnets, EC2 instances, and load balancers to manage the high volume of FASTag processing.
Planned the use of Kubernetes for container orchestration, ensuring efficient resource management and scaling.
Integrated deep learning models for document validation, ensuring accurate processing of vehicle-related documents.
Implementation:
Deployed the architecture on AWS, utilizing services such as EC2, S3, API Gateway, and Lambda for serverless computing.
Implemented a robust deployment pipeline using Docker, Jenkins, and Ansible, ensuring smooth deployment and updates to the infrastructure.
Guided the team in setting up FastAPI for API management and Celery for task distribution, ensuring efficient processing of FASTag validations.
Testing and Validation:
Conducted extensive testing of the architecture to ensure it could handle the expected load of 14,000 FASTags daily, along with the backlog.
Validated the security setup, ensuring all data transfers and storage met the required compliance standards.
Deployment and Monitoring:
Deployed the infrastructure on AWS, ensuring high availability and reliability through load balancing and auto-scaling features.
Implemented monitoring and logging using AWS CloudWatch and other tools to track performance and quickly address any issues.
Technologies Used
AWS Infrastructure:
Services: VPC, EC2, Secret Manager, API Gateway, IAM, Lambda, Load Balancers, S3
Purpose: Entire infrastructure hosted on AWS, managing networking, computing, storage, security, and serverless functions.
Containerization and Deployment:
Technologies: Docker, Jenkins, Ansible, Kubernetes
Purpose: Containerization and orchestration of applications for efficient deployment and scaling. Automated deployment pipeline for continuous integration and delivery.
Application Layer:
Technologies: FastAPI, Celery, Uvicorn, Gunicorn
Purpose: API management and task distribution for processing FASTag validations. FastAPI for building the web framework, Celery for asynchronous task management.
Machine Learning and Document Validation:
Technologies: Deep Learning Models
Purpose: Used for validating vehicle-related documents (RC, axle count, number plates) to ensure accurate KYV processing.
Monitoring and Security:
Technologies: AWS CloudWatch, IAM, Secret Manager
Purpose: Monitoring and managing security across the entire infrastructure, ensuring compliance and data protection.