Ruminating on FastAPI’s Speed and how to scale it with multiple Uvicorn workers

Python’s Global Interpreter Lock (GIL) often raises questions about concurrency and performance, especially for web frameworks like FastAPI. How does FastAPI stay so fast despite the GIL, and how can you run it with multiple workers to fully leverage multi-core CPUs? 

Let’s explore these concepts clearly.

The Global Interpreter Lock, or GIL, is a mutex that ensures only one thread executes Python bytecode at any given moment inside a single process. This simplifies memory management and protects Python objects from concurrent access issues. However, it means pure Python threads cannot run code in parallel on multiple CPU cores, limiting how multi-threaded Python programs handle CPU-bound tasks.

This sounds like bad news for a web framework that needs to handle many requests simultaneously, right? Not entirely.

How FastAPI Achieves High Performance Despite the GIL?

FastAPI is designed to handle many simultaneous requests efficiently by leveraging Python’s asynchronous programming capabilities, specifically the async/await syntax.

• Asynchronous I/O: FastAPI endpoints can be defined as async functions. When these functions perform I/O operations like waiting for a database query, network response, or file access, they yield control (using await) back to an event loop. This means while one request is waiting, the server can start working on other requests, without the need for multiple threads running in parallel.
• Single-threaded event loop: FastAPI runs on ASGI servers like Uvicorn that manage an event loop in a single thread. This avoids the overhead and complexity of thread locking under the GIL because only one thread executes Python code at a time, but efficiently switches between many tasks waiting for I/O.
• Ideal for I/O-bound tasks: Web APIs typically spend a lot of time waiting for I/O operations, so asynchronous concurrency lets FastAPI handle many requests without needing multiple CPU cores or threads.

But What If Your Application Is CPU-bound or You Need More Parallelism?

For CPU-bound workloads (heavy calculations) or simply to better utilize multi-core CPUs for handling many requests in parallel, you need multiple processes. This is where Uvicorn’s worker processes come in. 

Uvicorn, the ASGI server often used to run FastAPI, supports spawning multiple worker processes via the --workers option. Each worker is a separate process with its own Python interpreter and GIL. Workers run independently and can handle requests concurrently across different CPU cores. The master Uvicorn process listens on a port and delegates incoming requests to the worker processes.

This model effectively bypasses the single-thread GIL limitation by scaling workload horizontally over processes rather than threads (unlike multi-threading in Java or .NET frameworks - e.g. Spring Boot, ASP.NET MVC)

Set the number of workers roughly equal to your CPU cores for optimal utilization. Each worker is a separate process, so memory usage will increase.

When deploying with containers or orchestration tools like Kubernetes, it’s common to run one worker per container and scale containers horizontally.

Please NOTE that 95% of web applications and REST apis are NOT CPU-bound, but I/O bound. So even a single FastAPI server with async programming should more than suffice. Throw-in an additional server with a load balancer for high availability. 

But what if you have synchronous libraries and cannot run async in FastAPI? Well FastAPI can handle sync routes also as follows:

When FastAPI routes are defined as synchronous functions (def), the framework handles them by running the route handlers in an external thread pool instead of the main event loop thread. This approach prevents blocking the server's event loop, allowing requests to be processed concurrently despite the synchronous code. The synchronous route is effectively executed on a worker thread managed by the thread pool executor in the underlying Starlette framework. While this allows parallel execution, blocking I/O operations in sync routes still consume a thread and can reduce scalability under heavy load. Therefore, sync routes in FastAPI run concurrently but rely on thread-based parallelism rather than true asynchronous non-blocking concurrency as with async def routes. The default number of threads in FastAPI's thread pool for handling synchronous routes is 40. 

Ruminating on Continued Pre-Training and Fine-Tuning of LLMs

In my previous blogpost, we discussed about the differences between RAG and Fine-Tuning. Besides fine-tuning, there is another technique called as "Continued Pre-Training" that can be used to improve the performance of LLMs. 

Continued pre-training involves taking a pre-trained model—typically trained on a large, general dataset .....and further training it on a new, often domain-specific dataset. The goal is to adapt the model’s general knowledge to a specific domain, such as medical texts, legal documents, or scientific literature, without starting from scratch. This enhances the model’s understanding of a specific domain while retaining its general knowledge.

Suppose you have a pre-trained language model like BERT, originally trained on a general corpus like Wikipedia and BookCorpus. You want to use it for analyzing medical research papers. Since BERT’s general training may not capture medical jargon or context, you perform continued pre-training. 

To do this, you gather a large dataset of medical texts, such as PubMed articles or clinical notes. Fine-tune BERT’s weights on the medical corpus, allowing it to learn medical terminology and context. The new model (call it  “MedicalBERT”) has adapted to medical terminology and can better understand domain-specific texts.

Other examples of continued pre-training:

• Adapting a Language Model for Legal Documents: You have a pre-trained model like RoBERTa, trained on general web data, but you need it to understand legal terminology and context for analyzing contracts or court documents.
• Adapting a Vision Model for Satellite Imagery: A pre-trained vision model like ResNet, trained on ImageNet (general images like animals and objects), needs to be adapted for analyzing satellite imagery for urban planning or environmental monitoring.

Fine-tuning takes a pre-trained model (or a model after continued pre-training) and trains it on a smaller, task-specific dataset to optimize it for a particular task, such as classification, translation, or question answering. Fine-tuning adjusts the model’s weights to improve performance on the target task while leveraging the general knowledge learned during pre-training.

Examples of fine-tuning:

• Fine-Tuning for Object Detection in Medical Imaging: You want to use a pre-trained vision model like YOLOv5, adapted for medical imaging (e.g., via continued pre-training on X-ray images), to detect specific abnormalities like tumors in chest X-rays.

Given below is a comparison table for RAG vs Continued-Pretraining vs Fine tuning

Aspect	Retrieval-Augmented Generation (RAG)	Continued Pre-Training	Fine-Tuning
Definition	Combines a pre-trained language model with a retrieval mechanism to fetch relevant external documents for generating contextually accurate responses.	Further trains a pre-trained model on a large, domain-specific dataset to adapt it to a particular domain.	Optimizes a pre-trained model for a specific task using a smaller, labeled dataset in a supervised manner.
Objective	Enhance model responses by incorporating external knowledge dynamically during inference.	Adapt a model to understand domain-specific patterns, terminology, or context.	Optimize a model for a specific task, such as classification or translation.
Data Requirement	Requires a large corpus of documents for retrieval (often unstructured) and a pre-trained model.	Requires a large, domain-specific dataset, typically unlabeled or weakly labeled.	Requires a smaller, task-specific, labeled dataset.
Learning Type	Combines generative modeling with retrieval; no additional training required during inference.	Self-supervised or unsupervised learning (e.g., masked language modeling).	Supervised learning with task-specific objectives (e.g., classification loss).
Process	Retrieves relevant documents from an external knowledge base and uses them as context for the model to generate responses.	Continues training the model on domain-specific data to update its weights broadly.	Updates model weights specifically for a target task using labeled data.
Computational Cost	Moderate; requires efficient retrieval systems but no additional training during inference.	High; involves training on large datasets, requiring significant compute resources.	Moderate to low; uses smaller datasets, but may require careful tuning to avoid overfitting.
Data Availability	Needs a well-curated, accessible knowledge base for retrieval (e.g., Wikipedia, company documents).	Requires a large, domain-specific corpus, which may be hard to obtain for niche domains.	Needs labeled data, which can be costly or time-consuming to annotate.
Model Modification	No modification to model weights; relies on external knowledge for context.	Broad updates to model weights to capture domain-specific knowledge.	Targeted updates to model weights for task-specific performance.
Scalability	Scales well with large knowledge bases, but retrieval quality affects performance.	Scales with data and compute resources; time-consuming for large datasets.	Scales with labeled data availability; risk of overfitting with small datasets.

Securing and validating JWT tokens

JSON Web Tokens (JWTs) are widely used for authentication and authorization in web applications. However, securely storing JWTs on the client side is critical to prevent vulnerabilities like cross-site scripting (XSS) attacks. While local storage and session storage are common choices, HttpOnly cookies offer a superior alternative for storing JWTs. 

HttpOnly cookies are browser cookies with the HttpOnly flag set, meaning they cannot be accessed by client-side JavaScript. This restriction makes them immune to XSS attacks, as malicious scripts injected into a webpage cannot read or manipulate the cookie's contents. Unlike local storage or session storage, which are fully accessible to JavaScript, HttpOnly cookies are managed by the browser and sent automatically with HTTP requests to the specified domain.

HttpOnly cookies can be configured with additional security flags like Secure and SameSite:

• Secure: Ensures the cookie is only sent over HTTPS, preventing interception over unencrypted connections.
• SameSite: Mitigates cross-site request forgery (CSRF) attacks by restricting when cookies are sent in cross-origin requests. For example, setting SameSite=Strict ensures the cookie is only sent for requests originating from the same site.

HttpOnly cookies allow developers to set precise expiration times, aligning with the JWT's own expiration. This ensures the token is automatically removed by the browser when it expires, reducing the risk of stale tokens lingering on the client.

To maximize the security of HttpOnly cookies for JWT storage, follow these best practices:

• Always set the HttpOnly, Secure, and SameSite attributes.
• Use short-lived JWTs with refresh tokens to minimize the impact of potential breaches.

Some security experts feel that CSRF tokens should be used along with cookies. 

But when using SameSite=Strict on cookies, CSRF tokens are generally not required for most scenarios because SameSite=Strict provides strong protection against cross-site request forgery (CSRF) attacks. 

Here's why:

• How SameSite=Strict Works: The SameSite=Strict attribute ensures that the browser only sends the cookie with requests originating from the same site (i.e., the same domain as the cookie). Cross-origin requests, such as those initiated by a malicious site, will not include the cookie, effectively blocking CSRF attacks.
• Why CSRF Tokens May Not Be Needed: Since the cookie (e.g., containing a JWT) is not sent with cross-origin requests, an attacker cannot exploit the cookie to make unauthorized requests on behalf of the user. This eliminates the primary vector for CSRF attacks, making additional CSRF tokens redundant in many cases.

If you cannot have the restriction of "SameSite=Strict", then it is best to implement CSRF tokens. Also for extremely secure mission-critical applications, having CSRF tokens will add an additional layer of security. 

Validation of JWT on server side:

Validating JWTs using signatures ensures the token's integrity and authenticity. This process relies on cryptographic signatures, which can be implemented using symmetric or asymmetric algorithms.

Symmetric key validation uses a single shared secret key for both signing and verifying the JWT. The most common algorithm is HMAC (e.g., HMAC-SHA256, denoted as HS256 in JWT). The recipient (e.g., a server) uses the same secret key to recompute the HMAC of the Header and Payload. If the computed signature matches the one in the JWT, the token is valid and untampered. Both the issuer (Identity JWT provider) and verifier (server side code) must share the same secret key securely.

So on the server-side, this entails storing the secret symmetric key securely in some place. 

Asymmetric key validation uses a pair of keys: a private key for signing and a public key for verification. Common algorithms include RS256 (RSA with SHA-256) and ES256 (ECDSA with SHA-256). The issuer (Identity JWT provider) signs the JWT using the private key. The recipient uses the corresponding public key to decrypt and verify the signature. If the decrypted hash matches the hash of the Header and Payload, the token is valid. The private key is kept secret by the issuer, while the public key can be freely distributed to verifiers.

For most modern, distributed web applications, asymmetric key validation (e.g., RS256) is preferred due to its flexibility and security, while symmetric key validation (e.g., HS256) suits simpler, internal systems.

Ruminating on "Zero ETL"

Traditional ETL workflows consist of three separate stages: 

1. Extracting data from source systems
2. Transforming it to fit analytical needs
3. Loading it into a target database or data warehouse. 

Although this method is reliable, it often requires considerable time, resources, and technical effort, which can introduce delays and impede timely decision-making.

In contrast, Zero ETL eliminates these conventional steps by allowing direct access to source data and applying transformations on-the-fly during query execution. This significantly reduces latency, limits unnecessary data movement, and streamlines the integration process. By harnessing modern cloud infrastructure and sophisticated query tools, Zero ETL offers a more efficient, scalable approach to data management.

Examples of Zero ETL:

• AWS’s Zero ETL solutions, such as the integration between Amazon Aurora and Amazon Redshift, allow organizations to query data across systems without constructing traditional pipelines.
• The Snowflake Data Cloud supports federated queries and data sharing, enabling access to data across platforms without ETL processes. 
• Google Cloud BigQuery Omni facilitates cross-cloud analytics, allowing users to query data residing in AWS, Azure, or Google Cloud Platform without data replication.
• Airbyte is a popular open-source data integration engine that automates the movement of data from various sources to destinations (data warehouses, lakes, databases) with minimal custom coding. Airbyte offers over 350 pre-built connectors, orchestration features, and robust security, making it suitable for streamlined, scalable data integration without heavy ETL pipelines.

Ruminating on AI Security

Artificial intelligence has evolved from a distant vision into a transformative force reshaping industries and daily life. Yet, alongside its immense potential lies a pressing need: security. AI systems, unlike conventional applications, bring distinct vulnerabilities that require a fundamental rethink of security strategies. 

Security by Design: A Core Principle, Not a Last Step

In AI, security isn’t a feature to tack on—it’s a foundational element that must permeate every phase of the process. From initial design to development, deployment, and ongoing management, a "secure by default" mindset ensures that protection is intrinsic to the system’s DNA, not an optional extra.

• Design: Establish clear security goals upfront, identifying potential risks and weaknesses.
• Development: Prioritize secure coding, rigorous testing, and techniques to strengthen models against attacks.
• Deployment: Implement safeguarded environments, strict access controls, and real-time monitoring.
• Operations: Maintain vigilance with ongoing assessments, monitoring, and rapid-response plans.

Effective AI security hinges on threat modeling—assessing how a breached AI component could ripple across systems, users, organizations, and society. Proactively imagining these scenarios sharpens our defenses. Consider risks like data leaks, operational collapses, or AI weaponization by bad actors.Recognize the ethical stakes, as insecure AI can amplify societal harm.

AI applications face threats that traditional security frameworks aren’t built to handle. Here are some critical challenges:

• Training Data Tampering: Attackers can poison datasets, skewing models to produce biased or dangerous results, risking flawed decisions or breakdowns.
• Prompt Manipulation: In generative AI, crafted inputs can hijack outputs, leading to erratic or harmful behavior—especially in systems driven by user interactions.
• Model Theft and Reverse-Engineering: Adversaries may extract or decode models, exposing proprietary logic or sensitive data.
• Adversarial Inputs: Subtle tweaks to inputs can trick models into errors, undermining reliability.

A secure AI future demands investment in key areas:

• Developer Empowerment: Equip teams with training in secure coding, responsible AI practices, and advanced techniques like adversarial hardening and privacy preservation.
• Thorough Monitoring: Deploy robust systems to log and track AI inputs—queries, prompts, or requests—ensuring accountability, auditability, and swift action if compromised.
• Collective Expertise: Encourage collaboration among researchers, developers, and security experts to pool insights and solutions.
• Proactive Audits: Regularly evaluate AI systems to uncover and patch vulnerabilities.

Securing AI is not a one-time fix but a continuous endeavor requiring relentless innovation and alertness. By embedding security across the AI lifecycle and tackling its unique challenges head-on, we can forge a dependable AI ecosystem that safely unlocks its promise for society.

What is an "AI Factory"?

 "AI Factory" is a metaphor for a scalable and industralized approach for building and deploying AI solutions in the enterprise. 

An AI factory builds intelligence the way a regular factory builds products. It takes data as its starting point, uses AI tools to process it, and creates smart outputs like predictions and automated tasks, all in a structured and repeatable way. 

Just like a factory turns materials into goods, an AI factory turns data into useful AI results for the enterprise.  "AI Factory" approach is imperative for creating AI models and applications in a standardized, repeatable way and incorporates all the best practices of DataOps, MLOps/GenOps to consistently deliver value to the business. 

The key pillars for an AI Factory are as follows:

• Scalable Data Platform: Automation of DataOps value chain - think of data pipelines and data governance. 
• AI Models & MLOps: Automation of model training, model versioning, scalable model deployment for inference, data drift measurement, model performance monitoring, etc. 
• Alignment with AI Stategy: As discussed here - https://www.narendranaidu.com/2025/02/crafting-ai-strategy-for-enterprise.html
• Seamless integration into business processes: This is about making AI a key part of the company's decision-making and operational processes. Applying AI's findings to directly influence and improve how the business runs. Embed AI to turbo-charge all automation activities.

While ad-hoc AI projects can be unpredictable, an AI factory delivers consistent and reliable results, minimizing errors. This reliability allows companies to innovate faster, adapt to market changes, and personalize offerings, ultimately outperforming competitors.

Ruminating on Data Mesh and Data-as-a-Product

In today’s fast-paced, data-driven world, organizations are constantly looking for better ways to handle the massive amounts of data they generate and use. Traditional techniques frequently rely on a single, centralized staff to manage all the data—like a gigantic control center managing the central data warehouse or data lake. But as firms develop and data grows more complicated, this traditional technique may become sluggish, wasteful, and impossible to scale. 

Data Mesh is a fresh and innovative strategy that’s altering how organizations think about data. Data Mesh is a decentralized way to manage data. Instead of one team being in charge of everything, Data Mesh spreads the responsibility across different groups—or "domains"—within the organization. A department such as product development, sales, or marketing could be considered a domain. Each domain owns its own data, meaning they collect it, store it, maintain its quality, and make it available to others. 

Data Mesh is built on a few key ideas:

• Domain-Driven Ownership: Each team takes full control of the data tied to their area of work. For example, the sales team manages sales data, while the customer support team handles support-related data.
• Self-Service: Domains get access to platforms and technology that let them manage their data independently, without always needing help from a central IT team. Even though data is managed separately, there are company-wide standards to make sure everything connects smoothly and stays secure.
• Data as a Product: One of the standout ideas in Data Mesh is treating data as a product. This concept is borrowed from how companies build software products—with a focus on making them user-friendly, reliable, and well-supported. In a Data Mesh, each domain doesn’t just store data, they polish it up and package it like a product that others in the organization can easily use.

The marketing team would build a ready-to-use "customer behavior data product," making it easily accessible via an API. This allows other departments, such as product design and leadership, to directly utilize reliable, well-organized data without needing to process raw data or request assistance.

Benefits of Data Mesh Architecture: Giving teams ownership of their data promotes scalability, allowing them to manage their own data needs as the company expands. It also accelerates workflows, as teams can independently develop and share data products. This ownership drives higher data quality, as teams rely on its accuracy, and provides the flexibility to adjust data to changing demands, leading to a more responsive organization.

Potential Challenges:While decentralization offers benefits, it necessitates careful coordination to prevent data silos and ensure interoperability. Managing numerous independent data products presents complexity, requiring teams to have adequate technical resources and skills. Robust governance is also crucial to avoid data duplication and security breaches.

For example - By using Data Mesh, an online retailer lets teams own their data: product manages catalogs, customer service handles reviews, and logistics oversees shipping. These "data products" are then easily accessible to other teams, like for a live sales dashboard, without needing a central data team, while maintaining consistency through shared standards.

Data Mesh is a mindset shift, not just technology, that empowers teams to own their data as products, unlocking its full potential, especially in large companies. Though requiring setup efforts, it leads to faster, smarter, and more adaptable data use.

Crafting an AI strategy for the enterprise

AI has emerged as a pivotal force for enterprise transformation, offering avenues to reduce operational costs, enhance service delivery, improve customer experiences, boost employee productivity, and generate new revenue streams. Given below is a simple structured approach that can be leveraged by enterprises to craft their AI strategy.

1. Define the business objectives: The foundation of any AI strategy lies in a clear articulation of the enterprise's business vision and goals. This step ensures that AI initiatives are not pursued in isolation but are deeply integrated with the company's strategic objectives.

• Clarifying the vision: The business vision should be a well-defined, inspiring direction that guides all strategic decisions. For instance, a retail enterprise might envision becoming the market leader in personalized shopping experiences by 2028. This vision sets the stage for AI applications that enhance customer interactions.
• Setting specific objectives: Objectives should be specific, measurable, achievable, relevant, and time-bound (SMART). Examples include increasing sales by 20% within a year or reducing customer service response times by 30%. These goals provide clear targets for AI to support, such as automating routine inquiries to free up human agents for complex issues.
• AI alignment: The AI strategy must align with these goals, identifying areas where AI can provide a competitive edge or address specific challenges. For example, if the goal is to enhance customer satisfaction, AI can be leveraged through chatbots for instant support or through personalized recommendation engines. This alignment ensures that AI efforts are not just technological experiments but strategic enablers.

2. Define success metrics and identify potential AI solutions: Once the business goals are established, the next step is to delineate how AI will drive these objectives and define metrics to measure success. This step is crucial for ensuring that AI initiatives deliver tangible business value.

• Identifying AI Use Cases: AI applications span a wide range, including predictive analytics for sales forecasting, automation for streamlining processes, customer segmentation for targeted marketing, and fraud detection for security. Identify and funnel the most relevant to your business objectives. 
• Mapping AI solutions to Goals: For each business goal, map the relevant AI use cases. If the goal is to reduce operational costs, AI can automate routine tasks like data entry, freeing up employees for higher-value work. If the goal is to improve customer experience, AI chatbots can handle inquiries, improving response times. This mapping ensures every AI project has a direct link to a strategic objective. 
• Defining Success Metrics: Success metrics should be quantifiable and aligned with business goals. For reducing costs, metrics could include the percentage reduction in manual labor hours, cost savings from automation, or improved process efficiency. For enhancing customer experience, metrics might include customer satisfaction scores, retention rates, or net promoter scores (NPS). For example, if the goal is to generate new revenue streams, success metrics could include the revenue generated from AI-powered products, such as subscription-based AI tools, or the increase in cross-sell opportunities through AI-driven recommendations. 

3. Define workstreams and Project Prioritization Framework:With a clear understanding of business goals and AI's role, the next step is to define specific workstreams (projects or initiatives) and prioritize them based on their potential impact and feasibility. This ensures efficient resource allocation and focus on high-value projects.

• Identify projects: List potential AI projects that align with the business goals and AI use cases identified earlier. For example, deploying AI for customer segmentation to improve marketing effectiveness or using AI for supply chain optimization to reduce costs. 
• Prioritization criteria: Develop a framework to prioritize these projects. Key criteria include:
• Business Impact: The potential value the project can bring, such as revenue growth or cost reduction.
• Technical Feasibility: The ease or difficulty of implementing the project, considering current technological capabilities.
• Resource Requirements: The resources (time, money, personnel) needed, ensuring alignment with available budgets and skills.
• Risk Assessment: The potential risks associated with the project, such as ethical concerns or technical challenges.
• Prioritization matrix: Use a matrix or scoring system to evaluate each project against these criteria and rank them accordingly. For example, assign scores from 1 to 5 for each criterion and calculate a total score for prioritization. A project with high business impact, low technical risk, and minimal resource requirements would rank higher. This systematic approach ensures that enterprises focus on initiatives with the greatest return on investment.

4. Address AI risks: AI implementation introduces risks such as data bias, privacy concerns, security vulnerabilities, and ethical dilemmas. Managing these risks through robust governance is essential for sustainable AI adoption.

• Risk identification: Common risks include algorithmic bias leading to unfair outcomes, data security breaches, privacy violations, and non-compliance with regulatory standards. 
• Mitigation strategies: Develop specific strategies to address these risks. To mitigate algorithmic bias, implement regular auditing of AI models for fairness and accuracy, using tools like AI Fairness 360. For data security, employ robust encryption and access control measures. To address privacy concerns, ensure compliance with regulations like GDPR. 
• Governance structures: Establish governance bodies or committees to oversee AI projects, set policies, and ensure compliance. This could include an AI ethics committee to review and approve AI models before deployment. Governance also involves training employees on AI ethics, implementing data governance policies, and conducting regular audits.

5. Establish AI Governance and MLOps:To sustain AI’s value, it is imperative to integrate governance with Machine Learning Operations (MLOps) for scalable, reliable systems.

• AI governance: Beyond risk mitigation, governance sets policies for AI lifecycle management development, deployment, and updates. This includes defining roles (e.g., data scientists, compliance officers) and standards for transparency and accountability. 
• MLOps framework: MLOps operationalizes AI by streamlining model training, deployment, monitoring, and maintenance. Tools like MLflow or Kubeflow automate workflows, ensuring models perform consistently in production. 
• Continuous monitoring: Track model performance (e.g., data drift) and business alignment, retraining as needed. For example, an AI chatbot’s effectiveness might decline if customer queries evolve, requiring updates.

With this structured approach for crafting an AI strategy, enterprises can unlock the full potential of AI, driving innovation, efficiency, and drive growth. 

Ruminating on Standardizing Data

In the realm of statistics, we frequently face datasets of varied sizes and units. This might make it difficult to compare variables or use specific statistical approaches. To solve this challenge, we use a strong approach known as standardization. 

Essentially, standardization transforms our original data into a new dataset where:

• Mean:The average value of the new dataset is 0. 
• Standard Deviation:The measure of data dispersion around the mean is 1.

This process is also known as "z-score transformation".

Below are the advantages of standarizing data: 

• Comparability: Standardized data enables direct comparison of variables recorded on various scales. For example, heights in meters can be compared to weights in kilos.
• Model Development: Standardized data improves the performance of many statistical models, including regression and machine learning methods. This increases the model's accuracy and stability.
• Outlier Detection: When data is normalized, it is easier to identify numbers that vary considerably from the norm.

The formula for standardizing a data point (x) is: 

z (standard value) = (x - mean) / standard-deviation

Example:

• Original data: 150, 160, 170, 180, 190
• Mean (μ) = 170, Standard Deviation (σ) = 15.8
• Standardized data: -1.27, -0.63, 0, 0.63, 1.27

Standardizing data is a fundamental technique in statistics and data science. By transforming data to have a mean of 0 and a standard deviation of 1, we gain valuable insights and improve the performance of various statistical analyses.

Top 5 GenAI usecases in the industry

There are hundreds of GenAI usecases in the market today. But if we were to summarize them under some common themes, then the following five areas are the overarching themes that most usecases will fall under. 

1. Revolutionizing Customer Service Automation:

Imagine a world where customer service is available 24/7, responds instantly, and provides personalized solutions without long wait times. This is the promise of Generative AI in customer service. GenAI-powered chatbots and virtual assistants are moving beyond simple keyword matching to engage in natural, human-like conversations. They can handle a wide array of customer service tasks, including:

• Answering FAQs: Instead of relying on pre-written scripts, these AI assistants can understand the nuances of customer questions and provide dynamic, context-aware answers.
• Providing Product Information: GenAI can access and process vast amounts of product data to offer detailed descriptions, comparisons, and personalized recommendations.
• Troubleshooting Technical Issues: By understanding the technical context of a problem, these AI systems can guide customers through troubleshooting steps, often resolving issues without human intervention.
• Processing Orders and Returns: From order tracking to return authorizations, GenAI can streamline these processes, providing a seamless customer experience.

This automation not only improves customer satisfaction but also frees up human agents to handle more complex or sensitive issues, resulting in a more efficient and cost-effective customer service operation.

2. Supercharging Developer Productivity:

Generative AI is becoming an indispensable tool for developers, significantly boosting their productivity. By automating repetitive tasks and providing intelligent assistance, GenAI is changing the software development landscape. Key benefits include:

• Code Generation and Completion: Tools like GitHub Copilot use GenAI to suggest code completions, generate entire functions from natural language descriptions, and even create boilerplate code, drastically reducing coding time.
• Automated Testing and Debugging: GenAI can generate test cases, identify potential bugs, and even suggest fixes, leading to higher-quality and more reliable software.
• Simplified Documentation: GenAI can automatically generate documentation from code, ensuring that documentation is always up-to-date and reducing manual effort.

By handling mundane tasks, GenAI allows developers to focus on more creative and complex problem-solving, leading to faster development cycles and innovative solutions.

3. Enhancing Personal Productivity with AI Copilots:

Beyond software development, GenAI is also empowering individuals to boost their personal productivity. AI "copilots" are emerging as valuable assistants for various tasks:

• Email and Document Drafting: GenAI can help write emails, create reports, and even draft presentations, saving time and improving the quality of written communication.
• Task Management and Scheduling: GenAI can analyze schedules, prioritize tasks, and even suggest optimal times for meetings or focused work.
• Idea Generation and Brainstorming: GenAI can act as a brainstorming partner, generating new ideas, exploring different perspectives, and helping overcome creative blocks.

These AI copilots are not meant to replace human effort but rather to augment it, allowing individuals to work smarter, not harder.

4. Transforming Content Creation Across Formats:

Generative AI is revolutionizing content creation across various formats:

• Graphics: Tools like DALL-E 2 and Midjourney can create stunning visuals from simple text descriptions, opening up new possibilities for graphic design and visual storytelling.
• Video: GenAI is being used to create short videos, animations, and even generate realistic synthetic media, transforming video production and entertainment.
• Text: From writing marketing copy and blog posts to generating creative fiction and poetry, GenAI is empowering writers and content creators with new tools and capabilities.

This democratization of content creation is empowering individuals and businesses to create high-quality content more efficiently and at scale.

5. Revolutionizing Knowledge Management:

The sheer volume of information available today can be overwhelming. GenAI is helping us manage and utilize this information more effectively:

• Summarization and Synthesis: GenAI can condense large amounts of text into concise summaries, making it easier to grasp key information quickly.
• Knowledge Extraction and Organization: GenAI can extract key concepts and relationships from unstructured data, helping to organize and structure knowledge for easier access and retrieval.
• Personalized Learning and Recommendations: GenAI can analyze individual learning styles and preferences to recommend relevant resources and create personalized learning paths.

By making information more accessible and manageable, GenAI is empowering individuals and organizations to make better decisions and drive innovation.

Reducing IT costs - A three-pronged strategy

In today's digital age, IT costs can quickly spiral out of control. To maintain profitability and competitiveness, businesses must proactively seek ways to reduce these expenses.

1. Harnessing the Power of Automation

• Hyper Automation: By automating repetitive tasks across IT and business processes, organizations can increase efficiency, reduce errors, and free up valuable resources.
• GenAI-Driven Knowledge Search: Implementing a GenAI-powered knowledge search solution can empower employees to find answers quickly and accurately, reducing support costs and improving productivity.
• Automated Asset Management: An automated asset management system like ServiceNow can streamline IT operations, reduce manual tasks, and improve asset visibility.

2. Optimize Cloud Spend

• FinOps: Adopting a FinOps approach allows organizations to manage cloud costs effectively by combining financial accountability, engineering practices, and business objectives.
• Automated DevSecOps/MLOps/GenOps: Reduce the number of resources required to manage your cloud operations. 
• IaaC automation: Infrastructure as Code automation can help in quickly provisioning and tear down of environements leading to cost savings.
• Leverage Dev Containers: Reduce onboarding time for developers and projects. 

3. Simplifying and Consolidating

• Rationalizing Enterprise Architecture: By simplifying your back office and eliminating redundant systems, you can reduce maintenance costs and improve overall efficiency.
• Consolidating Vendor Landscape: Reducing the number of vendors can lead to better pricing, improved service levels, and reduced administrative overhead.

Articulating the business value of IT solutions

When you're trying to sell an IT solution to your stakeholders, it’s super important to talk about more than just the tech specs. You need to show how it can actually help the business. 

Here’s a breakdown of five key ways to highlight the business value of your IT proposal:

1. Cost Reduction: Look for specific areas where your IT solution can help save money, like making processes smoother, automating tasks, or using resources more efficiently. Use solid metrics and financial forecasts to illustrate how much money could be saved. For instance, calculate potential savings from lower labor costs, reduced energy use, or less waste. Point out how these cost savings can lead to better profits, more competitiveness, and sustainable growth. Also emphasize on how these cost savings can create a "self-funding" model for their new transformative initiatives. 


2. Generating new Revenue Streams: Talk about how the IT solution can open up fresh revenue opportunities, like launching new products or services, tapping into new markets, or coming up with innovative business models. Share data that shows demand in the market and how profitable these new revenue streams could be (market share). Explain how the solution can grow with the business as it expands (scalability).


3. Enhancing Customer Experience: Get to know what problems customers face and how your IT solution can help solve them through a deep understanding of business processes. Describe how the solution will make customers happier through quicker responses, personalized services, or added convenience. Stress how a great customer experience can set your business apart from competitors and build customer loyalty. Try to articulate the ROI in terms of NPS improvement or CSAT improvements. 


4. Supporting Net Zero Goals of the Enterprise: Look at how the IT solution benefits the environment, like using less energy or cutting down carbon emissions. Align with the sustainability goals of the customer. Show how your solution fits into the company’s sustainability plans and helps create a greener future. Highlight the positive effects on society and local communities.


5. Gaining Deeper Business Insights: Explain how the IT solution can pull useful insights from current data, like spotting trends or opportunities. This is especially true when you are building data modernization solutions. Show how these insights can lead to smarter business choices based on data-driven-decisions. Emphasize that data-driven insights can give you an edge over competitors and spark innovation.

By clearly explaining the business value of ourIT solution,we have a better shot at getting buy-in from stakeholders and ensuring a successful rollout. 

Ruminating on JDK builds and versions

Ever since Oracle changed the licensing of Java in 2019, there has been a lot of confusion in the market. Found the below blog article to be extremely useful in this regard - 

https://www.marcobehler.com/guides/a-guide-to-java-versions-and-features

As of today, the best bet for developers looking for LTS (Long Term Support) is to use Eclipse Temurin. Memebers of this consortium include IBM/Red Hat, Microsoft, Azul, New Relic, Alibaba, T-systems, etc. A list of all OpenJDK builds by Temurin is available here https://adoptium.net/temurin/releases/

Other option for LTS is Amazon Corretto: https://aws.amazon.com/corretto/?filtered-posts.sort-by=item.additionalFields.createdDate&filtered-posts.sort-order=desc. Amazon provides LTS at no extra cost. Similary Microsoft, Red Hat also provide their LTS bundles of OpenJDK. 

It is important to note that OpenJDK available at https://jdk.java.net/23/ (openjdk.org) does not have LTS. OpenJDK.org site is managed by Oracle primarily and Oracle provides TLS only if you purchase the license of Oracle Java SDK (not OpenJDK). 

So, if you are just using the latest download from Openjdk site, then as soon as a new version is available, the older versions are NOT patched! So you will not get security patches for older versions and this can be a concern for production environments. 

OpenJDK builds having LTS are typically supported with patches for 4 years...so you have time to plan your upgrades. Security is a major concern in software development. LTS versions receive regular security updates, ensuring that vulnerabilities are patched in a timely manner. Another good blog showcasing the various LTS options is here - https://www.petefreitag.com/blog/java-lts-versions/ and https://adoptium.net/support/

Ruminating on DORA regulation

The Digital Operational Resilience Act (DORA), a new EU regulation, aims to strengthen the cybersecurity and operational resilience of financial institutions and their critical ICT providers. IT companies, particularly those serving the financial sector, must be ready to comply with DORA's comprehensive requirements by its enforcement date of January 17, 2025.

DORA is basically a set of rules for financial companies in the EU to make sure they're safe from cyberattacks and other tech problems. It's like a safety net to keep their services running smoothly, no matter what happens. This applies to banks, insurance companies, and even the tech companies that help them out.

DORA's Core Components are as follows: 

• Cyber Risk Management Framework: Organizations must establish a comprehensive plan for identifying, assessing, and mitigating risks related to their information and communication technology systems.
• Incident Response and Reporting Systems: Entities are required to implement procedures for monitoring, detecting, and reporting ICT-related incidents.
• Digital Operational Resilience Testing: Regular testing of ICT systems is mandatory to evaluate their resilience against cyber threats and operational disruptions.
• Third-Party Risk Management Controls: Stricter measures are necessary to assess and manage the risks associated with outsourcing ICT services to third-party providers.
• Information Sharing Mechanisms: Entities must participate in collaborative efforts to share intelligence and best practices regarding cyber threats.

To comply with DORA regulations, an enterprise is expected to do the following: 

1. Check Your Risk Management Plan: First, see if your organization already has a plan for managing ICT risks. This plan should include rules, procedures, and regular checks that fit your organization's specific risks.
2. Identify Gaps: Look at your current plan and compare it to what DORA requires. Find any areas where you might be lacking, like security testing or managing risks from third-party vendors. This will help you know what changes you need to make.
3. Review Your Incident Response: Make sure your processes for handling incidents are strong enough to meet DORA's standards. This means you should be able to watch for, manage, and report incidents effectively.
4. Improve Testing Procedures: Update your testing plan to include regular checks for vulnerabilities and penetration tests. DORA requires that critical organizations conduct threat-led penetration testing (TLPT) every three years.
5. Manage Third-Party Risks: Put in place strict measures for handling risks from third-party service providers. This includes keeping a detailed list of all contracts with these providers.
6. Share Information: Set up ways to share information about cyber threats with other organizations in the financial industry. Working together can help everyone become more resilient against cyber threats.

Ruminating on AWS Fargate Autoscaling

Amazon Fargate is a serverless compute engine that allows you to run containers without having to provision or manage servers. One of its powerful features is automatic scaling, which enables your application to adjust its capacity based on demand. This ensures optimal performance and cost efficiency.

Target Tracking and Step Scaling: A Dynamic Duo

Fargate automatic scaling primarily relies on two strategies: target tracking and step scaling. Let's delve into how these mechanisms work together to maintain desired application performance.

Target Tracking:

• Defining a Metric: You specify a metric that represents your application's performance or resource utilization. This could be CPU utilization, memory usage, or a custom metric.
• Setting a Target Value: You establish the desired target value for the metric. For instance, you might set a target CPU utilization of 70%.
• Continuous Monitoring: Fargate continuously monitors the actual metric value and compares it to the target.
• Scaling Actions: If the actual value deviates significantly from the target, Fargate triggers scaling actions to adjust the number of tasks.

Step Scaling: 

• Step Adjustments: Step scaling involves increasing or decreasing the number of tasks by a predefined step size.
• Scaling Policies: You define scaling policies that specify:
• Step size: The number of tasks to add or remove in each scaling action.
• Cooldown period:The minimum time between scaling actions to prevent excessive fluctuations.
• Thresholds:The deviation from the target metric that triggers scaling.

How They Work Together:

• Target Tracking: Fargate monitors the specified metric and determines if it's deviating from the target.
• Step Scaling: If the deviation exceeds the defined thresholds, Fargate applies the corresponding scaling policy.
• Adjustment: The number of tasks is increased or decreased by the step size.
• Evaluation: Fargate continues to monitor the metric and adjusts the number of tasks as needed to maintain the target value.

Imagine a web application/API that experiences sudden traffic spikes during peak hours. By using target tracking and step scaling, you can configure Fargate to automatically increase the number of tasks when demand surges, ensuring optimal performance for your users.

Ruminating on INVEST principles for user stories

The User Story, a concise, informal explanation of a desired product or functionality from the user's perspective, is a key component of Agile techniques. To ensure that these user stories are valuable, clear, and actionable, a set of principles known as INVEST were developed. 

The INVEST acronym, coined by Bill Wake, outlines six essential qualities that user stories should possess:

• Independent: User stories should be self-contained and not rely on others. This ensures that they may be built and tested independently, eliminating dependencies and simplifying the development process.
• Negotiable: User stories are not contracts. They should be open to discussion and negotiation with stakeholders to ensure that they align with business objectives and user needs.
• Valuable: User stories should deliver tangible value to the user or business. They should address real pain points or provide new capabilities that enhance the user experience.
• Estimable: User stories should be estimable in terms of effort and time required to complete them. This enables the development team to plan effectively and prioritize work.
• Size: User stories should be small enough to be completed within a single iteration or sprint. This promotes a steady flow of work and prevents the team from becoming overwhelmed with large, complex tasks.
• Testable: User stories should be testable to ensure that they meet the defined acceptance criteria. This helps to verify that the implemented functionality meets the user's expectations. 

To ensure that your user stories adhere to the INVEST principles, consider the following guidelines:

• Prioritize independence: Break down large, complex features into smaller, independent user stories.
• Foster negotiation: Encourage open communication and collaboration with stakeholders to refine user stories and ensure they align with business objectives. 
• Focus on value: Identify user stories that directly address customer needs or provide significant business benefits. 
• Estimate effort: Use techniques like story points or planning poker to estimate the relative size of user stories.
• Define acceptance criteria: Clearly articulate the conditions that must be met for a user story to be considered complete.
• Maintain size: Keep user stories small and focused to avoid overwhelming the development team.

By following the INVEST principles, we can create user stories that are clear, actionable, and aligned with the overall goals of our project. This will help improve communication, increase productivity, and deliver higher-quality software.

Calculating tokens for words

For LLM applications, we often use embedding models like ada-002 or davinci models. While using these models, we need to often estimate the number of tokens that would required for our application. 

For the English language, a good thumb rule is that 3 to 4 chars make up a token. 

A nifty online tool that can help you estimate the number of tokens is: https://www.quizrise.com/token-counter

Database Manager - DBGate

 If you are looking for a web-based database manager that is open-source and commercial friendly, then please have a look at DBGate: https://github.com/dbgate/dbgate

Since it is web-based, you can see a good demo here: https://demo.dbgate.org/

#no-build for JS: Embracing the Era of Import Maps and HTTP/2

Complex build processes have dominated the web development world for years. JavaScript module bundling, transpiling, and management have become critical tasks for tools such as Webpack. However, what if there was an easier method? 

It is time for us to "un-learn" old tricks and rid ourselves of the baggage of old knowledge! Modern browsers support HTTP/2 and "import maps" for javascript modules and this is an absolute game changer!

In the past, web browsers had trouble comprehending contemporary JavaScript features and had trouble loading many small files quickly. Webpack and other build tools addressed these problems by:

• Bundling: Putting several JavaScript files together into a single, bigger file to cut down on HTTP requests.
• Transpiling:transforming current JavaScript syntax into an earlier iteration that works with an earlier browser.

Although these technologies were helpful to us, they increased complexity:

• Build Configuration: Build procedures frequently cause disruptions to the development workflow, necessitating continuous rebuilding and waiting. 
• Development Workflow Disruption: Build configurations can be time-consuming to set up and manage.

But all modern browsers (Chrome, Firefox, Safari, Edge) support HTTP/2 and "Import Maps" for JS modules. 

• Import maps enables us to use alias for full JS module paths - it is a method of creating a central registry that associates module names with their real locations is. This makes things clearer and more versatile by removing the requirement for you to write whole file paths in your code (across multiple files). 
• HTTP/2 is a more effective and rapid method of sending data across the internet. It makes it unnecessary to bundle files because browsers can handle several tiny files well. Instead of opening many connections to the server (like having many waiters running around), HTTP/2 uses one connection. Thus, multiple JS files can be downloaded concurrently, speeding up page load times.

Should we serve JS files from a CDN?

An excellent article that states why we do not need to actually use a CDN for serve popular JS libraries:  https://blog.wesleyac.com/posts/why-not-javascript-cdn

Excerpt from the article: 

"This means that one of these CDNs going down, or an attacker hacking one of them would have a huge impact all over society — we already see this category of problem with large swaths of the internet going down every time cloudflare or AWS has an outage."

Ruminating on Core Web Vitals

Have you ever clicked on a webpage only to spend a long time staring at a blank screen? Yes, it is frustrating. That's bad for the website (since you could just click away) and bad for you (because you're waiting).

This is where Core Web Vials are useful -- they are a set of metrics defined by Google to measure a website’s loading speed, interactivity and visual stability.  In essence, there are three things that websites must do well in order to ensure that users have a positive experience. 

• Quick loading (also known as Largest Contentful Paint, or LCP): This refers to how quickly a webpage's major content loads. A decent load time is defined as 2.5 seconds or less. 
• Smooth interactions (First Input Delay or FID): This is about how responsive a webpage feels. If you click a button and nothing happens for a while, that's a sign of bad FID. We want those clicks to feel instant, just like if you were pushing a real button. A decent speed is one that is less than 100 milliseconds.
• Stable visuals (Cumulative Layout Shift or CLS): This one's about how much the content on a webpage jumps around as it loads. Imagine you start reading a recipe and then all the ingredients suddenly jump to different places on the page - that's bad CLS! We want the content to stay put so you can focus on what you're reading. A score of under 1.0 is good.

More information on Core Web Vitals can be found here -- https://developers.google.com/search/docs/appearance/core-web-vitals

The top recommentations to improve the core web vitals are as follows:

• Optimize Image size and Image loading: Large, poorly optimized pictures are a primary cause of sluggish loading speeds.  Reducing the size of your pictures, using compression techniques, and turning on lazy loading—which loads images only when the user scrolls—will all help you get a higher Largest Contentful Paint (LCP) score. Also implement lazy loading - i.e. make images load only when visitors scroll down to them. 
• Caching:  Enable browser caching to store website elements on a user's device.  This way, the browser doesn't have to download everything all over again each time they visit your site. Leveraging a CDN would also help here. 
• JS & CSS optimization: Minify and compress your CSS and Javascript files.  This can significantly reduce their size and improve loading times.
• Preload: Preloading instructs the browser to fetch specific resources early, even before they're explicitly requested by the page. You can preload resources using the <link rel="preload"> tag in the <head> section of your HTML document.

Gaia-X & Catena-X: Data usage governance and Sovereignty

Unless you have been living under a rock, you must have heard about Gaia-X. The whole premise of Gaia-X was to build a fair and open data ecosystem for Europe, where everyone can share information securely and control who gets to use it. It's like a big marketplace for data, but with European values of privacy and control at its heart.

The core techical concepts that we need to understand around Gaia-X are "Data Spaces" and "Connectors".  

Data Spaces refer to secure and trusted environments where businesses, individuals, and public institutions can share and use data in a controlled and fair manner. They're like marketplaces for data, but with strict rules to ensure privacy, security, and data sovereignty (meaning individuals and companies retain control over their data).

A connector plays a crucial role in facilitating secure and controlled data exchange between different participants within the ecosystem. Think of it as a translator and bridge builder, helping diverse systems and providers communicate and share data seamlessly and safely. The Eclipse foundation has taken a lead on this and created the Eclipse Dataspace Component (EDC) initiative wherein many opensource projects have been created to build Gaia-X compliant connectors. 

These core concepts of Dataspaces and Connectors can also be used to build a modern data architecture in a federated decentralized manner. An excellent article on this approach is available on AWS here - https://aws.amazon.com/blogs/publicsector/enabling-data-sharing-through-data-spaces-aws/

An offshoot of Gaia-X is another initiative called Catena-X that aims to create a data ecosystem for the automotive industry. It aims to create a standardized way for car manufacturers, suppliers, dealers, software providers, etc. – to share information securely and efficiently through usage of standard data formats and procotols. The Eclipse Tractus-X™ project is the official open-source project in the Catena-X ecosystem under the umbrella of the Eclipse Foundation and has reference implementations of connectors to securely exchange data. 

But how do you ensure that the data is used only for the purpose that you allowed it to be used? Can you have legal rights and controls over how the data is used after you have shared it? This is the crux of the standards around Gaia-x/Catena X. 

At the heart lies the data usage contract, a legally binding agreement between data providers and consumers within the Catena-X ecosystem. This contract specifies the exact terms of data usage, including:

• Who can access the data?: Defined by roles and permissions within the contract.
• What data can be accessed?: Specific data sets or categories permitted.
• How the data can be used?: Allowed purposes and restrictions on analysis, processing, or sharing.
• Duration of access?: Timeframe for using the data.

Contracts establish a basic link between the policies and the data to be transferred; a transfer cannot occur without a contract. 

Because of the legal binding nature of this design, all users of data are required to abide by the usage policies just like they would with a handwritten contract. 

More details around data governance can be found in the official white paper --https://catena-x.net/fileadmin/_online_media_/231006_Whitepaper_DataSpaceGovernance.pdf

Besides contracts, every data access and usage event is logged on a distributed ledger, providing a transparent audit trail for accountability and dispute resolution. The connectors also enforce proper authentication/authorization through the Identity Provider and validate other policy rules. 

Thus Gaia-X/Catena-X enforce data usage policies through a combination of legal contracts, automated technical tools, independent verification, and a strong legal framework. This multi-layered approach ensures trust, transparency, and accountability within the data ecosystem, empowering data providers with control over their valuable information.

Generating synthetic data

 Faker is an excellent tool for generating mock data for your application. But any complex application would have tens of tables with complex relationships between them. How can we use Faker to populate all of these tables? 

We can follow two approaches here:

Option 1: Create the primary table first and then the dependent tables. Then when populating the dependent tables, you can refer to a random primary key from the first table. A good article summarizing this is here -- https://khofifah.medium.com/how-to-generate-fake-relational-data-in-python-and-getting-insight-using-sql-in-bigquery-985c5adc87d3

Code snippet: 

 #generate relational user id in account table and transaction table

trans['user_id']=random.choices(account["id"], k=len(trans))

Option 2: Use a ORM framework to insert data into the database. An ORM framework would make it easy to establish relationships between different tables. A good article on this approach is here - https://medium.com/@pushpam.ankit/generating-mock-data-for-complex-relational-tables-with-python-and-sqlite-2950ab7700f2

Another interesting opensource tool is "Synthetic Data Vault" https://sdv.dev/

In these tools, we first train the tool on real data and then use the AI model for generation of new synthetic data. Many vendors differentiate between "mock" and "synthetic" data on this aspect. 

Fine-tuning vs RAG for LLMs

Large language models (LLMs) have revolutionized the field of natural language processing (NLP), enabling state-of-the-art performance on a wide range of tasks, including text classification, translation, summarization, and generation. 

When it comes to usecases around leveraing LLMs for extracting insights from our own knowledge repositories, we have broadly two design approaches:

• Fine-tuning a LLM
• RAG (Retrieval Augmented Generation)

Many fields have their own specialised terminology. This vocabulary may be missing from the common pretraining data utilised by LLMs. 

Fine Tuning a LLM

The process of fine-tuning a pre-trained LLM on a fresh domain-specific dataset is known as fine-tuning. Fine-tuning is the process of further training a previously trained LLM on a smaller, domain-specific, labelled dataset.

To fine-tune an LLM, you'll need a dataset of labelled data, with each data point representing an input and output pair. A written passage, a query, or a code snippet might be used as input. The result might be a label, a summary, a translation, or code.

Once you have a dataset, you can use a supervised learning method to fine-tune the LLM. By minimising a loss function, the algorithm will learn to map the input to the output.

It can be computationally costly to fine-tune an LLM.

Another subset of the above approach is called PEFT (Parameter-efficient fine-tuning) and LoRA is the most popular approach for PEFT today.  

LoRA (Low-Rank Adaptation of Large Language Models) is a fine-tuning approach for LLMs that is more efficient and memory-efficient than standard fine-tuning. Traditional fine-tuning entails altering all of an LLM's parameters. This can be computationally expensive and memory-intensive, particularly for big LLMs with billions of parameters.LoRA, on the other hand, merely modifies a few low-rank matrices. Because of this, LoRA is far more efficient and memory-efficient than conventional fine-tuning.

An excellent article explaining the concepts of full fine-tuning and LoRA is here -- https://deci.ai/blog/fine-tuning-peft-prompt-engineering-and-rag-which-one-is-right-for-you/

RAG (Retrieval Augmented Generation)

RAG is an effective strategy for improving the performance and relevance of LLMs by combining "prompt engineering" with "context retrieval" from external data sources.

Given below is a high level process flow for RAG. 

1. All documents from a domain specific knoweldge source are converted into embeddings and stored in a special vector database. These vector embeddings are nothing but “N-dimensional matrices” of numbers.
2. When the user types his query, even the query is converted into an embedding (matrix of numbers) using a AI model.
3. Semantic search techniques are used to identify all contextual sentences in the “document-embedding” for the given query. Most popular algorithm is “cosine similarity”. This algorithm uses a cosine maths function to get all the sentences (matrices) that are ‘near’ or ‘close’ to the query (matrix). This entails matrix multiplications and other maths functions. 
4. All retrieved “semantically similar sentences/paragraphs” from multiple documents are finally again sent to a LLM for ‘summarization’. The LLM would paraphrase all the disparate sentences into a coherent story that is readable. 

RAG along with Prompt Engineering can be used to build powerful knowlege management platforms such as this - https://www.youtube.com/watch?v=lndJ108DlBs

The table belows shows the advantages/disadvantages of both the approaches. For most usecases, a proper utilization of prompt engineering and RAG would suffice. 

Ruminating on Debezium CDC

Debezium is a distributed open source platform for change data capture (CDC). It collects real-time changes to database tables and transmits them to other applications. Debezium is developed on top of Apache Kafka, which provides a dependable and scalable streaming data infrastructure.

Debezium operates by connecting to a database and watching for table updates. When a change is identified, Debezium creates a Kafka event with the change's information. Other applications, such as data pipelines, microservices, and analytics systems, can then ingest these events.

There are several benefits of utilising Debezium CDC, including:

• Debezium feeds updates to database tables in near real time, allowing other applications to react to changes almost quickly.
• Debezium is built on Apache Kafka, which provides a dependable and scalable streaming data platform.
• Debezium can stream updates to a number of databases, including MySQL, PostgreSQL, Oracle, and Cassandra using connectors. 
• Debezium is simple to install and operate. It has connectors for major databases and may be deployed on a number of platforms, including Kubernetes/Docker.

Use cases for Debezium CDC:

• Data pipelines and real-time analytics: Debezium can be used to create data pipelines that stream changes from databases to other data systems, such as data warehouses, data lakes, and analytics systems.  For example, you could use Debezium to stream changes from a MySQL database to Apache Spark Streaming. Apache Spark Streaming can then process the events and generate real-time analytics, such as dashboards and reports.

• Data synchronization: Debezium can be used to synchronize data between different databases or data systems.
• Microservices integrations: Communicate with downstream microservices using Event Driven Architecture paradigm - https://debezium.io/blog/2019/02/19/reliable-microservices-data-exchange-with-the-outbox-pattern/

Mock data and APIs

Mocking APIs and synthetic mock data generation are invaluable techniques to speed up development. We recently used the Mockaroo platform and found it quite handy to generate dummy data and mock APIs. 

https://www.mockaroo.com/

IBM has also kindly released ~25M records of synthetic financial transacation data that can be used during application development or ML training.

https://github.com/IBM/TabFormer

Other examples of mock data generation tools are:

• https://github.com/brianvoe/gofakeit
• https://faker.readthedocs.io/en/master/

Leveraging Graph Databases for Fraud Detection

 There are many techniques for building Fraud Detection systems. It can be:

• Rule Based (tribal knowledge codified)
• Machine Learning (detect anomalies, patterns, etc.)

There is a third technique using Graph Databases such as Neo4J, TigerGraph or Amazon Neptune

A graph network can assist identify hidden aspects of transactions that would otherwise be missed just by looking at data in a relational table.

Lets consider the example of indenfying fraud in a simple financial transaction. Every financial transaction has thousands of attributes associated with is - e.g. amount, IP address, browser, OS, cookie data, bank, geo-location, card details, recepient,etc.

Using a graph database, we can build a graph network where each transaction is a node and the line connections (aka edges) represent the attributes of the transaction. The following article gives a good primer on how this kind of network would look like - https://towardsdatascience.com/fraud-through-the-eyes-of-a-machine-1dd994405e6e

Once the graph is created, there are many techniques that can be used to detect patterns and relationships between the different attributes. 

• Link Analysis: This approach is used to detect unusual links between network items. In a financial network, for example, you may check for linkages between accounts engaged in fraudulent activities.
• Anomaly detection: This approach is used to identify entities or transactions that differ from usual behaviour in a network. In a credit card network, for example, you may watch for transactions performed from strange areas or for abnormally big sums.
• Cluster Analysis:  This technique is used to identify groups of entities in a network that are closely connected to each other. Clustering may also be used to surface commercial ties or social circles in a transaction banking graph.

Thus, by employing graph analytics, we may detect clusters and links in their data, revealing previously unknown possible fraud connections. More information on such techniques can be found on this blog: https://www.cylynx.io/blog/network-analytics-for-fraud-detection-in-banking-and-finance/

Because of their capacity to track complicated chains of transactions, graph databases are particularly useful in financial crime use cases and fraud detection graph analysis. Traditional RDBMS struggle with these sequence of connections because multiple recursive inner joins are necessary to accomplish this sort of traversal query in SQL, which is very challenging. 

A few articles that give good illustrations on this topic:

https://www.graphable.ai/blog/graph-database-fraud-detection/

https://towardsdatascience.com/fraud-detection-with-graph-analytics-2678e817b69e

https://neo4j.com/blog/enterprise-fraud-detection/?ref=web-solutions-fraud-detection