Shadow testing is a software testing technique that involves sending production traffic to a duplicate or shadow environment. This allows testers to compare the behavior of the new feature in the shadow environment to the behavior of the old feature in the production environment. This can help to identify any potential problems with the new feature before it is released to all users.
The following diagram from the Microsoft GitHub site illustrates this concept.
Differential privacy (DP) is a mathematical paradigm for protecting individuals' privacy in datasets. By allowing data to be analysed without disclosing sensitive information about any individual in the dataset, it can protects the privacy of individuals. Thus, it is a method of protecting the privacy of people in a dataset while maintaining the dataset's overall usefulness.
To protect privacy, the most easy option is anonymization, which removes identifying information. A person's name, for example, may be erased from a medical record. Unfortunately, anonymization is rarely enough to provide privacy because the remaining information might be uniquely identifiable. For example, given a person's gender, postal code, age, ethnicity, and height, it may be able to identify them uniquely even in a massive database.
The concept behind differential privacy is to introduce noise into the data in such a manner that it is hard to verify whether any specific individual's data was included in the dataset. This is accomplished by assigning a random value to each data point, which is chosen in such a manner that it has no effect on the overall statistics of the dataset but makes identifying individual data points more difficult.
The following paper by Apple gives a very good overview of how Apple uses Differential Privacy to gain insight into what many Apple users are doing, while helping to preserve the privacy of individual users - https://www.apple.com/privacy/docs/Differential_Privacy_Overview.pdf
Epsilon (ε) is a parameter in differential privacy that affects the amount of noise introduced to the data. A greater epsilon number adds more noise, which gives more privacy but affects the accuracy of the findings.
Here are some examples of epsilon values that might be used in different applications:
nip.io is a free, open-source service that allows you to use wildcard DNS for any IP address. This implies you may build a hostname that resolves to any IP address, no matter where it is. This may be beneficial for a number of things, including:
It is more critical than ever in today's complex and dispersed IT settings to have a complete grasp of how your systems are performing. This is where the concept of observability comes into play. The capacity to comprehend the condition of a system by gathering and analysing data from various sources is referred to as observability.
Observabilty has three critical pillars:
Distributed Logging: Logs keep track of events that happen in a system. They may be used to discover problems, performance bottlenecks, and the flow of traffic through a system. In a modern scalable distributed architecture, we need logging frameworks that support collection and ingestion of logs across the complete tech stack. Platforms such as Splunk and ELK (Elastic, Logstash, Kibana) support this and are popular frameworks for distributed logging.
Metrics (performance instrumentation in code): Metrics are numerical measures of a system's status. They may be used to monitor CPU use, memory consumption, and request latency, among other things. Some of the most popular frameworks for metrics are Micrometer , Prometheus and DropWizard Metrics.
Tracing (E2E visibility across the tech stack): Traces are a record of a request's route through a system. They may be utilised to determine the core cause of performance issues and to comprehend how various system components interact with one another. A unique Trace-ID is used to corelate the request across all the components of the tech stack.
Platforms such as Dynatrace, AppDynamics and DataDog provide comprehensive features to implement all aspects of Observability.
The three observability pillars operate together to offer a complete picture of a system's behaviour. By collecting and analysing data from all three sources, you can acquire a thorough picture of how your systems operate and discover possible issues before they affect your consumers.
There are a number of benefits to implementing the three pillars of observability. These benefits include:
If you want to increase the observability of your systems, I recommend that you study more about the three pillars of observability and the many techniques to apply them. You can take your IT operations to the next level if you have a thorough grasp of observability.
There has been a lot of buzz in recent years about the potential of large language models (LLMs) to develop new text forms, translate languages, compose various types of creative material, and answer your queries in an instructive manner. However, one of the drawbacks of LLMs is that they may be quite unexpected. Even little changes to the prompt might provide drastically different outcomes. This is where quick engineering comes into play.
The technique of creating prompts that are clear, explicit, and instructive is known as prompt engineering. You may maximise your chances of receiving the desired outcome from your LLM by properly writing your questions.
Given below are some of the techniques you can use to create better prompts:
If you have an OpenAPI specs file (YAML or JSON), then you can quickly create a mock server using one of the following tools.
Today Kubernetes has become the defacto standard to deploy applications. To understand what happens behind the scenes when you fire "kubectl" commands, please have a look at this excellent tutorial series by VMWare - https://kube.academy/courses/the-kubernetes-machine
Some key components of the K8 ecosystem. The control plane consists of the API server, Scheduler, etcd and Controller Manager.
The following two articles by Rebecca Bevans are an excellent introduction to the concept of Hypothesis testing and the types of statistical tests available:
Snippet from the article on the process of hypothesis testing:
Step 1: State your null and alternate hypothesis
Step 2: Collect data
Step 3: Perform a statistical test
Step 4: Decide whether to reject or "fail to reject" your null hypothesis
Step 5: Present your findings
The following site has an excellent collection of 20 free courses that I would highly recommend for folks who want to learn the basics of finance and fundamentals of maths/stats in finance.
https://corporatefinanceinstitute.com/collections/?cost=free&card=117884
I really liked the following courses and helped me consolidate my understanding:
- Stats basics: https://learn.corporatefinanceinstitute.com/courses/take/statistics-fundamentals
- Accounting basics: https://learn.corporatefinanceinstitute.com/courses/take/learn-accounting-fundamentals-corporate-finance
- How to read financial statements: https://learn.corporatefinanceinstitute.com/courses/take/learn-to-read-financial-statements-free-course/
- Data Science fundamentals - https://learn.corporatefinanceinstitute.com/courses/take/data-science-and-machine-learning/
The following websites gives an excellent overview for beginners of the 3 different types of variables that we encounter in feature engineering (or even in basic stats):
https://study.com/academy/lesson/continuous-discrete-variables-definition-examples.html
https://www.scribbr.com/methodology/types-of-variables/
Snippets from the articles:
A discrete variable only allows a particular set of values, and in-between values are not included. If we are counting a number of things, that is a discrete value. A dice roll has a certain number of outcomes, and nothing else (we can roll a 4 or a 5, but not a 4.6). A continuous variable can be any value in a range. Usually, things that we are measuring are continuous variables, because it can be any value. The length of a car ride might be 2 hours, 2.5 hours, 2.555, and so on.
Categorical variables are descriptive and not numerical. So any way to describe something is a categorical variable. Hair color, gum flavor, dog breed, and cloud type are all categorical variables.
There are 2 types of categorical variables: Nominal categorical variables are not ordered. The order doesn't matter. Eye color is nominal, because there is no higher or lower eye color. There isn't a reason one is first or last.
Ordinal categorical variables do have an order. Education level is an ordinal variable, because they can be put in order. Note that there is not some exact difference between the levels of education, just that they can be put in order.
The TM Forum (TMF) is an organisation of over 850 telecom firms working together to drive digital innovation. They created a standard known as TMF Open APIs, which provides a standard interface for the interchange of various telco data.
TM Forum’s Open APIs are JSON-based and follow the REST paradigm. They also share a common data model for Telecom.
Any CSP (Communications service provider) can accelerate their API journey by leveraging the TMForum API contracts. The link below gives some of the examples of the API standards available:
https://projects.tmforum.org/wiki/display/API/Open+API+Table
Currently, there are around 60+ APIs defined in the Open API table of TMForum.
Few examples of the APIs are as follows:
The GitHub repository of TMForum is a great place to get acquainted with the APIs - https://github.com/tmforum-apis
Since the TMForum defines the data model in JSON format, any noSQL datastore that stores data as JSON documents becomes an easy option to quickly implement an API strategy. For example, TMF data model of the API can be persisted 1:1 in Mongo database without the need for additional mappings as shown here - https://www.mongodb.com/blog/post/why-telcos-implement-tm-forum-open-apis-mongodb
mAP (mean Average Precision) is a common metric used for evaluating the accuracy of object detection models. The mAP computes a score by comparing the ground-truth bounding box to the detected box. The higher the score, the more precise the model's detections.
The following articles give a good overview of the concepts of precision, recall, mAP, etc.
https://jonathan-hui.medium.com/map-mean-average-precision-for-object-detection-45c121a31173
https://blog.paperspace.com/mean-average-precision/
https://blog.paperspace.com/deep-learning-metrics-precision-recall-accuracy/
https://www.narendranaidu.com/2022/01/confusion-matrix-for-classification.html
Some snippets from the above article:
"When a model has high recall but low precision, then the model classifies most of the positive samples correctly but it has many false positives (i.e. classifies many Negative samples as Positive). When a model has high precision but low recall, then the model is accurate when it classifies a sample as Positive but it may classify only some of the positive sample.
Higher the precision, the more confident the model is when it classifies a sample as Positive. The higher the recall, the more positive samples the model correctly classified as Positive.
As the recall increases, the precision decreases. The reason is that when the number of positive samples increases (high recall), the accuracy of classifying each sample correctly decreases (low precision). This is expected, as the model is more likely to fail when there are many samples.
The precision-recall curve makes it easy to decide the point where both the precision and recall are high. The f1 metric measures the balance between precision and recall. When the value of f1 is high, this means both the precision and recall are high. A lower f1 score means a greater imbalance between precision and recall.
The average precision (AP) is a way to summarize the precision-recall curve into a single value representing the average of all precisions. The AP is the weighted sum of precisions at each threshold where the weight is the increase in recall.
The IoU is calculated by dividing the area of intersection between the 2 boxes by the area of their union. The higher the IoU, the better the prediction.
The mAP is calculated by finding Average Precision(AP) for each class and then average over a number of classes."
I was trying to help my kids understand the importance of deeply understanding a concept, instead of just remembering facts.
I found the knowledge pyramid of Bloom an excellent illustration to help my kids understand how to build skills and knowledge. The following article on Vanderbilt University site is a good read to understand the concepts - https://cft.vanderbilt.edu/guides-sub-pages/blooms-taxonomy/
In a distributed microservices environment, we do not have complex 2-phase commit transaction managers.
We need a simpler approach and we have design patterns to address this issue. A good article explaining this strategy is available here - https://docs.microsoft.com/en-us/azure/architecture/reference-architectures/saga/saga
The crux of the idea is to issue compensating transactions whenever there is a failure in one step of the flow. A good sample usecase for a banking scenario (with source code) is available here - https://github.com/Azure-Samples/saga-orchestration-serverless
I have always been a great fan of the 'Contract-First' API design paradigm. The Swagger (OpenAPI) toolset makes it very simple to design new API contracts and object schemas.
There is a Swagger Pet Store demo available here, wherein we can design API contracts using a simple YAML file: https://editor.swagger.io/
After the API contract has been designed and reviewed, we can quickly generate stubs and client code in 20 languages using Swagger Codegen.
In my previous blog posts, we had discussed about Data Lakes and Snowflake architecture.
Since Snowflake combines the abilities of a traditional data warehouse and a Data Lake, they also market themselves as a Data Lakehouse.
Another competing opensource alternative that is headed by the company databricks is called Delta Lake. Delta Lake provides ACID transactions, scalable metadata handling, and unifies streaming and batch data processing on top of existing data lakes.
A good comparison between Snowflake and databricks Delta Lake is available here: https://www.firebolt.io/blog/snowflake-vs-databricks-vs-firebolt
Enterprises who are embarking on their data platform modernization strategy can ask the following questions to arrive at a best fit choice:
One of the fundamental challenges in migrating SAP ECC to S/4 HANA is the lack of knowledge on the plethora of customizations done on legacy SAP ECC.
To address this challenge, there are two leading products in the market - ServiceNow Gekkobrain and Celonis.
Gekkobrain helps teams in getting a deeper understanding of custom code that was developed to extend the functionality of SAP. It analyses the custom code base and identifies relevant HANA issues and also try to fix the issue.
Gekkobrain flows is a component that uses ML to understand and visualize the business process. It can understand main flows, sub flows and deviations and help in identifying automation opportunities. This capability of Gekkobrain flows is also branded as a Process Mining Tool.
ServiceNow acquired Gekkobrain to integrate it with the NOW platform and leverage its workflow engine to automate workflows for customized SAP code (during HANA migrations).
Another popular tool that is used for during SAP modernization is the popular process mining tool called as Celonis. Celonis is used for retrieving, visualizing and analyzing business processes from transactional data stored by the SAP ERP systems.
When we connect on-prem applications to cloud services, we have two options - a) Connect to the cloud service provider through the internet OR b) setup a dedicated network link between your data centre and the cloud provider.
Setting up a private network connection between your site and the cloud provider enables extremely low latency, increased bandwidth and a more consistent network performance.
Many enterprises are adopting a multi-cloud strategy and in such cases also, it is imperative that there is no latency between service calls across the cloud providers. Hence it makes sense to leverage a SDCI (software defined cloud interconnect) to connect between the hyperscalers.
Such networking services are provided by hyperscalers themselves or by dedicated network companies. Given below are links to some of the providers in this space.
*It is important to note that the core bitcoin network/ledger does not use any encryption. It is all hashing as explained here - https://www.narendranaidu.com/2022/05/ruminating-on-proof-of-work.html
When it comes to symmetric encryption, the most common standards are Data Encryption Standards (DES) and Advanced Encryption Standards (AES).
When it comes to asymmetric encryption (public key cryptography), the dominant standard is RSA (Rivest-Shamir-Adleman). Almost all the digital certificates (HTTPS/SSL) issued used RSA as the encryption standard and SHA256 as the hashing algorithm.
Given below is a screenshot of a digital certificate of a random HTTPs site. You can see the encryption algorithm and Hash function mentioned in the certificate.
ECC has the following advantages when compared to RSA:
In the bitcoin network, miners have to compare the hash value during the 'Proof-of-Work' process.
The target hash value is stored in the header and is expressed as a 67-digit number. Miners must find a new hash of the transaction block that is below the given target.
To solve the hash puzzle, miners will try to calculate the hash of a block by adding a nonce to the block header repeatedly until the hash value yielded is less than the target.
But how is the value of the hash calculated? In the bitcoin network, the value of a hash is calculated as follows:
Many folks get confused between the AWS terminology of 'Dedicated Instance' vs 'Dedicated Host'.
A simple way to understand the difference is to remember that a "host" is a physical machine that can host many virtual machine instances.
Hence a "dedicated host" is a physical machine that is dedicated to your organization. On this physical machine (host), you can install many VMs/containers. So you control what VMs (instances) are going to run on that host.
So what is a dedicated instance then? A dedicated instance is a virtual machine that runs on hardware that is not shared with other accounts. Dedicated instances are physically isolated at the host hardware level from instances that belong to other AWS accounts. Hence you can only be certain that the underlying hardware that is hosting your VM is not shared with someone else. But you have no fine-grained control over which VM would be launched on which host, etc.
Temenos is the world's number one core banking platform. It is built entirely on the AWS cloud and uses all managed services.
I was stuck with the simplicity of the overall architecture on AWS and how it enabled elastic scalability to scale-out for peak loads and also scale-back for reducing operational costs.
Another interesting aspect was the simple implementation of the CQRS pattern to offload queries (read-only API requests) to DynamoDB. The pipeline was built using Kinesis and Lambda.
An excellent short video on the AWS architecture of the Temenos platform is here: https://youtu.be/mtZvA7ARepM
In neural nets, we have to specify the number of epochs while we train the model.
One Epoch is defined as the complete forward & backward pass of the neural net over the complete training dataset. We need to remember that Gradient Descent is an iterative process and hence we need multiple passes (or epochs) to find the optimal solution. In each epoch, the weights and biases are updated.
Batch size is the number of records in one batch. One Epoch may consist of multiple batches. The training dataset is split into batches because of memory space requirements. A large dataset cannot be fit into memory all at once. With increase in Batch size, required memory space increases.
Iterations is the number of batches needed to complete one epoch. So if we have 2000 records and a batch size of 500, then we will need 4 iterations to complete one epoch.
If the number of epochs are low, then it results in underfitting the data. As the number of epochs increases, more number of times the weight are changed in the neural network and the curve goes from underfitting to optimal to overfitting curve. Then how do we determine the optimal number of epochs?
It is done by splitting the training data into 2 subsets - a) 80% for training the neural net b) 20% for validating the model after each epoch. The fundamental reason we split the dataset into a validation set is to prevent our model from overfitting. The model is trained on the training set, and, simultaneously, the model evaluation is performed on the validation set after every epoch.
Then we use something called as the "early stopping method"- we essentially keep training the neural net for an arbitrary number of epochs and monitor the performance on the validation dataset after each epoch. When there is no sign of performance improvement on your validation dataset, you should stop training your network. This helps us arrive at the optimal number of epochs.
A good explanation of these concepts is available here - https://www.v7labs.com/blog/train-validation-test-set. Found the below illustration really useful to understand the split of data between train/validate/test sets.
Convolutional Neural Nets (CNNs) have made Computer Vision a reality. But to understand CNNs, we need to get basics right - What exactly is a convolution? What is a kernel/filter?
The kernel or filter is a small matrix that is multiplied by the source image matrix to extract features. So you can have a kernel that identifies edges or corners of a photo. Then there could be kernels that detect patterns - e.g. eyes, stripes.
A convolution is a mathematical operation where a kernel (aka filter) moves across the input image and does a dot product of the kernel and the original image. This dot product is saved as a new matrix and is called as the feature map. An excellent video visualizing this operation is available here - https://youtu.be/pj9-rr1wDhM
Image manipulation software such as Photoshop also use kernels for effects such as 'blur background'.
One fundamental advantage of the convolution operation is that if a particular filter is designed to detect a specific type of feature in the input, then applying that filter systematically across the entire input image allows us to discover that feature anywhere in the image. Also note that a particular convolutional layer can have multiple kernels/filters. So after the input layer (a single matrix), the convolutional layer (having 6 filters) will produce 6 output matrices. A cool application to visualize this is here - https://www.cs.ryerson.ca/~aharley/vis/conv/flat.html
A suite of tens or even hundreds of other small filters can be designed to detect other features in the image. After a convolutional layer, we also typically add a pooling layer. Pooling layers are used to downsize the features maps - keeping the important parts and discarding the rest. The output matrices of the pooling layer are smaller in size and faster to process.
So as you can see, the fundamental difference between a densely connected layer and a convolutional layer is that dense fully connected layers learn global patterns (involving all pixels) whereas convolution layers learn local features (edges, corners, textures, etc.)
Using CNNs, we can create a hierarchy of patterns - i.e. the second layer learns from the first layer. CNNs are also reusable, so we can take an image classification model trained on https://www.image-net.org/ dataset and add additional layers to customize it for our purpose.
A good introduction to CNN models is given in this article - https://towardsdatascience.com/convolution-neural-networks-a-beginners-guide-implementing-a-mnist-hand-written-digit-8aa60330d022 with a good PyTorch implementation for MNIST dataset here - https://towardsdatascience.com/mnist-handwritten-digits-classification-using-a-convolutional-neural-network-cnn-af5fafbc35e9
Activation functions play an important role in neural nets. An activation function transforms the weighted sum of the input into an output from a node.
Simply put, an activation function defines the output of a neuron given a set of inputs. It is called "activation" to mimic the working of a brain neuron. Our neurons get activated due to some stimulus. Similarly the activation function will decide which neurons in our neural net get activated.
Each hidden layer of a neural net needs to be assigned an activation function. Even the output layer of a neural net would use an activation function.
The ReLU (Rectified Linear Unit) function is the most common function used for activation.
The ReLU function is a simple function: max(0.0, x). So essentially it takes the max of either 0 or x. Hence all negative values are ignored.
Other activation functions are Sigmoid and Tanh. The sigmoid activation function generates an output value between 0 and 1.
An excellent video explaining activation function is here - https://youtu.be/m0pIlLfpXWE
The activation functions that are typically used for the output layer are Linear, Sigmoid (Logistic) or Softmax. A good explanation of when to use what is available here - https://machinelearningmastery.com/choose-an-activation-function-for-deep-learning/
Gradient Descent is the most popular optimization algorithm used to train machine learning models - by minimizing the cost function. In deep learning, neural nets use back-propagation that internally use a cost function (aka lost function) like Gradient Descent.
The Gradient descent function essentially uses calculus to find the direction of travel and then to find the local minimal of a function. The following 2 videos are excellent tutorials to understand Gradient Descent and their use in neural nets.
Once you understand these concepts, it will help you also realize that there is no magic involved when a neural net learns by itself -- ultimately a neural net learning by itself just means minimizing a cost function (aka loss function).
Neural nets start with random values for their weights (of the channels) and biases. Then by using the cost function, these hundreds of weights/biases are shifted towards the optimal value - by using optimization techniques such as gradient descent.
