API mock servers from OpenAPI specs

 If you have an OpenAPI specs file (YAML or JSON), then you can quickly create a mock server using one of the following tools. 

• Microrocks: https://microcks.io/documentation/using/openapi/
• Prism: https://docs.stoplight.io/docs/prism/674b27b261c3c-overview
• OpenAPI mock: https://github.com/muonsoft/openapi-mock
• Imposter: https://www.imposter.sh/

A list of all other OpenAPI tools is given here: https://openapi.tools/

Ruminating on TMForum

The TM Forum (TMF) is an organisation of over 850 telecom firms working together to drive digital innovation. They created a standard known as TMF Open APIs, which provides a standard interface for the interchange of various telco data.

TM Forum’s Open APIs are JSON-based and follow the REST paradigm. They also share a common data model for Telecom.

Any CSP (Communications service provider) can accelerate their API journey by leveraging the TMForum API contracts. The link below gives some of the examples of the API standards available: 

https://projects.tmforum.org/wiki/display/API/Open+API+Table

Currently, there are around 60+ APIs defined in the Open API table of TMForum. 

Few examples of the APIs are as follows:

• Customer Bill Management API: This API allows operations to find and retrieve one or several customer bills (also called invoices) produced for a customer.
• Customer Management API: Provides a standardized mechanism for customer and customer account management, such as creation, update, retrieval, deletion and notification of events.
• Digital Identity Management API: Provides the ability to manage a digital identity. This digital identity allows identification of an individual, a resource, or a party Role (a specific role - or set of roles - for a given individual).
• Account Management API: Provides standardized mechanism for the management of billing and settlement accounts, as well as for financial accounting (account receivable) either in B2B or B2B2C contexts.
• Geographic Address Management API: Provides a standardized client interface to an Address management system. It allows looking for worldwide addresses
• Geographic Site Management API: Covers the operations to manage (create, read, delete) sites that can be associated with a customer, account, service delivery or other entities.
• Payment Management API: The Payments API provides the standardized client interface to Payment Systems for notifying about performed payments or refunds.
• Payment Method Management API: This API supports the frequently-used payment methods for the customer to choose and pay the usage, including voucher card, coupon, and money transfer.
• Product Ordering Management API: Provides a standardized mechanism for placing a product order with all the necessary order parameters.
• Promotion Management API: Used to provide the additional discount, voucher, bonus or gift to the customer who meets the pre-defined criteria.
• Recommendation Management API: Recommendation API is used to recommend offering quickly based on the history and real-time context of a customer.
• Resource Function Activation Management API: This API introduces Resource Function which is used to represent a Network Service as well as a Network Function.

The GitHub repository of TMForum is a great place to get acquainted with the APIs - https://github.com/tmforum-apis

Since the TMForum defines the data model in JSON format, any noSQL datastore that stores data as JSON documents becomes an easy option to quickly implement an API strategy. For example, TMF data model of the API can be persisted 1:1 in Mongo database without the need for additional mappings as shown here - https://www.mongodb.com/blog/post/why-telcos-implement-tm-forum-open-apis-mongodb

API contract first design

 I have always been a great fan of the 'Contract-First' API design paradigm. The Swagger (OpenAPI) toolset makes it very simple to design new API contracts and object schemas. 

There is a Swagger Pet Store demo available here, wherein we can design API contracts using a simple YAML file: https://editor.swagger.io/ 

After the API contract has been designed and reviewed, we can quickly generate stubs and client code in 20 languages using Swagger Codegen. 

Automating API creation over existing data-sources

 If you need to quickly build REST APIs over existing data in excel sheets, RDBMS, NoSQL data stores; then the DreamFactory toolkit is immensely helpful. 

DreamFactory is available under the Apache 2 license and you can host it on your public/private cloud platforms. Just with a few clicks, you will have a complete secure API service available to perform CRUD operations on your data. DreamFactory runs on the Apache LAMP stack ((Linux, Apache, MySQL, PHP/Perl/Python). 

Besides generating the APIs, DreamFactory can also dynamically generate a Client SDK for HTML5 frameworks like jQuery, AngularJS, and Sencha, and a code library for native clients like iOS, Windows 8, and Android.

DreamFactory also has a toolkit to convert your SOAP webservices into REST APIs - https://www.dreamfactory.com/resources/video/how-turn-any-soap-web-service-rest-api/

It also has a cool feature called DataMesh wherein we can create virtual foreign key relationships between tables in the same database or between completely different databases without altering your schema or writing any code. Create, read, update, or delete objects and related objects with a single API call.

DreamFactory also automatically creates Swagger API documentation and has basic API Management capabilities built in (e.g. rate limiting)

Ruminating on Asynchronous Request-Reply pattern over HTTP

Quite often, a HTTP request would entail processing on some back-end that communicates via messaging. In such cases, do we keep the server thread waiting for a response on the queue? or do we have a better design pattern to handle such scenarios.

The following article on Microsoft illustrates a good pattern for Asynchronous Request-Reply pattern over HTTP - https://docs.microsoft.com/en-us/azure/architecture/patterns/async-request-reply

Excerpts from the article: 

1. The client sends a request and receives an HTTP 202 (Accepted) response.
2. The client sends an HTTP GET request to the status endpoint. The work is still pending, so this call also returns HTTP 202.
3. At some point, the work is complete and the status endpoint returns 302 (Found) redirecting to the resource.
4. The client fetches the resource at the specified URL.

Ruminating on Consumer Driven Contracts

One of my teams has successfully implemented the paradigm of consumer driven contracts in a recent digital transformation program. We were very happy with the Pact framework and the OOTB integration available in Java (Spring Boot).

Anyone still not convinced on using Consumer Driven Contracts should peruse the below link without fail - https://docs.pact.io/faq/convinceme

The fundamental advantage of  Consumer Driven Contracts is that only parts of the API that are actually used by the consumer get tested. Hence any provider behavior not used by current consumers is free to change without breaking tests.
When you are about to change the contract of a popular API, you can quickly check which consumers would be affected and where to focus your efforts on.

It is important to remember that Pact should not be used for functional testing...it is to be used only for contract adherence. Pact can also be integrated into your CI-CD process, wherein you run all the consumer contracts as part of the build. 

Ruminating on Jackson JSON Parsing

In my previous post, we discussed about how we can extract an arbitrary JSON value out of a JSON string. Very often, developers face another error while using Jackson - com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field error.

This happens when your JSON string has an attribute that is not present in your POJO and you are trying to deserialize it. Your POJO might not be interested in these fields or these fields could be optional.

To resolve this error, you have two options:

Option 1:  Disable the error checking on the Jackson Mapper class as follows.
objectMapper.disable(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES)

Option 2: If you have access to the POJO object, then you can annotate it as follows
@JsonIgnoreProperties(ignoreUnknown = true)

Timeout for REST calls

Quite often, we need a REST API to respond within a specified time frame, say 1500 ms.
Implementing this using the excellent Spring RestTemplate is very simple. A good blog explaining how to use RestTemplate is available here - http://www.baeldung.com/rest-template

   

Generating secure keys for encryption

Very often, we need to create secure keys that can be used in digital signatures or for signing a JWT.
It is very important to create a secure key so that the encryption is strong. A good post about this is here - https://docs.oracle.com/cd/E19424-01/820-4811/aakfw/index.html

Jotting down some snippets from the article:

The strength of encryption is related to the difficulty of discovering the key, which in turn depends on both the cipher used and the length of the key. 

Encryption strength is often described in terms of the size of the keys used to perform the encryption: in general, longer keys provide stronger encryption. Key length is measured in bits.

Different ciphers may require different key lengths to achieve the same level of encryption strength

------------------------------------------------------------------------
A key is nothing but a byte array that is passed to the encryption algorithm.  Hence if you are storing the key in a properties file, please makes sure that you have stored it in Base64 format.

Spring provides very simple methods for generating secure keys as shown below.

Database connection reconnection strategy

In any database connection pool, there is a risk of stale connections due to network outages or database server restarts. How do we refresh the connection pool without restarting the client application?

The standard technique used is to plug-in a validation query; that gets fired every time a connection is requested from the pool. This validation query is typically a default test query that does not result in any IO / disk access.  Examples of validation queries for different databases are given below:

• Oracle - select 1 from dual
• SQL Server - select 1 (tested on SQL-Server 9.0, 10.5 [2008])
• Postgresql - select 1
• hsqldb - select 1 from INFORMATION_SCHEMA.SYSTEM_USERS
• derby - values 1
• H2 - select 1
• DB2 - select 1 from sysibm.sysdummy1
• Mysql - select 1

We were using Spring Boot that uses the Tomcat JDBC Connection Pool by default. We tried setting all the parameters required for the validation check as given here, but in vain. 

Finally we decided to go with HikariCP connection pool as suggested here. 

First, we added the dependency in our maven pom as shown below. 

Next we added the following properties in our application.properties file. We will pick up these properties to create our HikariCP connection. pool.
Finally we wrote a @Configuration bean to wire up the HirakiCP datasource as below:

Spring Sleuth Tips and Tricks

Spring Sleuth coupled with Zipkin in an excellent tool to measure the performance bottlenecks in your distributed microservices architecture.

Spring Sleuth automatically creates HTTP and Messaging interceptors when calls are made to other microservices over REST or RabbitMQ. But what if you want to instrument the time taken to execute a method or time take for a third-party service to respond?

Initially we tried with the @NewSpan annotation on methods, but soon realized that this annotation DOES NOT work in the same class. A detailed discussion thread on this limitation is available here - https://github.com/spring-cloud/spring-cloud-sleuth/issues/617.

The core reason is that the proxies that Spring creates do not work for the method calls in the same object. Sleuth creates a proxy of the object so if we are calling a proxied method from the proxy itself then the method will be called directly without going through the proxy logic, hence no span will be created. Hence we need to manually add the span as follows and it would show up in the beautiful Zipkin UI.

Another great blog post around these techniques is here - http://www.baeldung.com/spring-cloud-sleuth-single-application

Ruminating on the single-threaded model of NodeJS and Node-RED

Many developers and architects have asked me questions on the single-threaded nature of NodeJS and whether we can use it effectively on multi-core machines. Since Node-RED is based on NodeJS, whatever is valid for NodeJS is also valid for Node-RED.

NodeJS has a single thread per process model. When you start NodeJS, it would start a single process with one thread in it. Due to it's non-blocking IO paradigm, it can easily handle multiple client requests concurrently, as there is no thread that is blocking for any IO operation.

Now the next question is around the optimal usage of multi-core machines. If you have 8 core or 16 core machines on your cloud and just run one NodeJS process on it, then you are obviously under-utilizing the resources you have paid for. In order to use all the cores effectively, we have the following options in NodeJS:

1) For compute-intensive tasks: NodeJS can fork out child processes for heavy-duty stuff - e.g. if you are processing images or video files, you can fork out child NodeJS processes from the parent process. Communication between parent and child processes can happen over IPC.

2) For scaling REST APIs: You can start multiple NodeJS processes on a single server - e.g. if you have 8 cores, start 8 NodeJS processes. Put a load balancer (e.g. nginx) in front of these processes. You would anyways have some kind of load-balancing setup in your production environment and the same can be leveraged.

3) Cluster support in newer versions on NodeJS: In the latest versions of NodeJS, you have support for Clusters. Cluster enables us to start multiple NodeJS processes that all share the same server port - e.g. 8 NodeJS processes all listening to port 80.  This is the best OOTB option available today in NodeJS.

Hence it is indeed possible to effectively utilize NodeJS on multi-core machines. A good discussion on the same is available on StackOverflow here.

Another interesting question that is often asked is whether Java developers should jump ship and start development in NodeJS because of its speed and simplicity. Also, NodeJS evangelists keep harping about the fact that the single-threaded nature of Node removes the complexity of multi-threading, etc.

Here are my thoughts on this:

1) Today Java has first class support for non-blocking IO, similar to NodeJ. Take a look at the superb open-source  Netty library that powers all the cloud services at Apple.

2) Most of the complexity of multithreading is abstracted away by popular frameworks such as Spring - e.g. Spring Cloud enables developers to write highly scalable and modular distributed applications that are cloud-native without dealing with the complexities of multi-threading.

3) The Servlet 3.0 specification introduced async requests and the Spring REST MVC framework also supports non-blocking REST services as described here.  

Thus today, all the goodies of NodeJS are available on the Java platform. Also plenty of  Java OSS to kick start your development with all the necessary plumbing code.

Ruminating on a nifty tool called 'ngrok'

Many a times developers want to test their APIs running on their development machine against a mobile app. To do so, developers typically do the following:

Option A) Create a hotspot Wifi on their development machine and connect the mobile phone to this Wifi. Since both the mobile phone and their development machine is on the same network, the mobile app can call the APIs running on the localhost of the developer machine.

Option B) We host the APIs on a public cloud (with a public IP), so that the mobile app can access it. For this, the developer needs to setup a automated CI/CD process that would deploy the REST APIs on the middleware.

There is a third option where-in, the developer can run the API server on his local machine and just create a proxy server on the internet that would securely tunnel the request to this local machine through the firewall. This can be done using ngrok - https://ngrok.com/

ngrok can be setup on your local machine within minutes and can serve as a great debugging tool. It has a network sniffer built in that can be assessed from http://localhost:4040. 

We would highly recommend this tool for any mobile/API developer. 

ngrok can also tunnel via SSH if you don't want to install or run the ngrok agent on the target machine - https://ngrok.com/docs/secure-tunnels/tunnels/ssh-reverse-tunnel-agent/

ngrok also has a nifty tool to setup a quick fileserver for files on your local machine. Just use the following command: ./ngrok http "file:///C:\Users\Docs" --basic-auth="user_id:password" 

API keys providing a false sense of security !

We have seen so many API implementations wherein an API key is the only thing used to secure APIs. API keys are typically long alphanumeric strings that give a false sense of security.

The entire onus of protecting that key and making only SSL requests lies with the API consumer. This is a very concerning, since this rarely happens. We have decompiled Android APKs and found API keys stored in config files. We have seen API keys checked-into source control systems such as GitHub :)

Kristopher Sandoval has written an excellent blog post on the prevalent usage of using API keys to secure your APIs.

We must not rely solely on API keys to secure our APIs, but rather use open standards such as OAuth 2, OpenID Connect, etc. to secure access to our APIs. Many developers use insecure methods of storing API keys in mobile apps or pushing the API key to Github.

Snippets from the article (http://nordicapis.com/why-api-keys-are-not-enough/) -

"Most developers utilize API Keys as a method of authentication or authorization, but the API Key was only ever meant to serve as identification.
API Keys are best for two things: identification and analytics (API metrics).

If an API is limited specifically in functionality where “read” is the only possible command, an API Key can be an adequate solution. Without the need to edit, modify, or delete, security is a lower concern."

Another great article by NordicAPIs is on the core concepts of Authentication, Authorization, Federation and Delegation - http://nordicapis.com/api-security-the-4-defenses-of-the-api-stronghold/
The next article demonstrates how these 4 core concepts can be implemented using OAuth and OpenID Connect protocols - http://nordicapis.com/api-security-oauth-openid-connect-depth/

Open Source API Management Tools

For folks, who are interested in setting up their own API Management tools, given below are a few options:

• Kong API Management - https://getkong.org/
• WSO2 API Management - http://wso2.com/products/api-manager/
• APIMan by JBoss - http://www.apiman.io/

Whitepaper on APIs - Digital Glue in the new economy

Recently I coauthored a whitepaper on APIs and the important role they play in the digital economy.

Link - http://www.syntelinc.com/insights-and-resources/publications/api-management-platforms-digital-glue-api-economy

Ruminating on SemVer

Semantic Versioning (aka SemVer) of components has become mainstream today. The official page laying out the guidelines is available here - http://semver.org/

Following SemVer, each component has a 3 digit version in the format of 'Major.Minor.Patch' - for e.g. 2.3.23

• You increment the major version, when you make incompatible changes. 
• You increment the minor version, when you make changes but those changes are backward compatible.
• The patch digit is incremented when you just make a bug-fix and it is obviously backward compatible.
• With SemVer, pre-releases can be defined by appending a hyphen and the word 'alpha/beta' after it. For e.g. a pre-release for version 3.0.0 could be 3.0.0-alpha.1. 

Following SemVer is a boon in managing dependencies between components. So if component A is using version 4.2.3 of component B, then you know that as long as version B does not become 5.x.y, there would be no breaking changes. You can specify dependencies in the manifest file of a component.

While using SemVer for software components is fine, does it make sense to have the x.y.z version in the URL of public APIs?
APIs are the interfaces you expose to your consumers. Do your consumers really need to know about the bug fixes you have made? or the new features you have added? Maybe yes or no !
IMHO, just using a single version number in your API URL would suffice majority of real life business usecases. For e.g. https://api.company.com/v1/customer.

A good blog post by APIGEE on API versioning is available here. As stated in the blog - "Never release an API without a version and make the version mandatory."

If you want to constrain the amount of information that you want back from the API (e.g. mobile client on slow networks), then you can follow this strategy. Another alternative is to look at new options such as GraphQL. 

APIs in Fleet Management

Fleet Management software is used by fleet owners to manage their moving assets. The software enables them to have a centralized data-store of their vehicle and driver information and also maintain maintenance logs (service and repair tracking).

The software also allows us to schedule preventive maintenance activities, monitor fuel efficiency, maintain fuel card records, calculate metrics such as "cost per mile" etc. You can also setup reminders for certification renewals and license expiration.

It was interesting to see Fleetio (a web based fleet management company) roll out a API platform for their fleet management software. Their vision is to become a digital hub for all fleet related stuff and turn their software product into a platform that can be leveraged by partners to create a digital ecosystem.

The API would allow customers to seamlessly integrate data in Fleetio with their operational systems in real time. For e.g. Pulling work orders from your fleet management system and pushing it to your accounting software in real time. Pushing mileage updates from a bespoke remote application to your fleet management software, Integrate driver records with Payroll systems, etc. All the tedious importing and exporting of data is gone !

TomTom also has a web based fleet management platform called as WEBFLEET that provides an API (Webfleet.connect) for integration. The Fleetlynx platform also has an API to integrate with Payroll and Maintenance systems.

Ruminating on the Open Bank Project

A colleague of mine introduced me to the 'Open Bank Project'. It's an interesting open source project to create a generic API layer on top of various core banking products. Third-party developers would then use these APIs to build cool mobile and social apps.

The open source project was started by a Berlin based company named Tesobe. Right now, they seem to have successfully built adapters/connectors to 3 German banks and are planning to add connectors for more banks. A full list is given here - https://api.openbankproject.com/connectors-status/

A few sample apps have been built using the Open Bank API - https://openbankproject.com/apps/

I think the concept is interesting, but the challenge would be to build the connectors to the various core banking products out there. Will download the APIs from github and evaluate them further.