Calculating tokens for words

For LLM applications, we often use embedding models like ada-002 or davinci models. While using these models, we need to often estimate the number of tokens that would required for our application. 

For the English language, a good thumb rule is that 3 to 4 chars make up a token. 

A nifty online tool that can help you estimate the number of tokens is: https://www.quizrise.com/token-counter

Database Manager - DBGate

 If you are looking for a web-based database manager that is open-source and commercial friendly, then please have a look at DBGate: https://github.com/dbgate/dbgate

Since it is web-based, you can see a good demo here: https://demo.dbgate.org/

#no-build for JS: Embracing the Era of Import Maps and HTTP/2

Complex build processes have dominated the web development world for years. JavaScript module bundling, transpiling, and management have become critical tasks for tools such as Webpack. However, what if there was an easier method? 

It is time for us to "un-learn" old tricks and rid ourselves of the baggage of old knowledge! Modern browsers support HTTP/2 and "import maps" for javascript modules and this is an absolute game changer!

In the past, web browsers had trouble comprehending contemporary JavaScript features and had trouble loading many small files quickly. Webpack and other build tools addressed these problems by:

• Bundling: Putting several JavaScript files together into a single, bigger file to cut down on HTTP requests.
• Transpiling:transforming current JavaScript syntax into an earlier iteration that works with an earlier browser.

Although these technologies were helpful to us, they increased complexity:

• Build Configuration: Build procedures frequently cause disruptions to the development workflow, necessitating continuous rebuilding and waiting. 
• Development Workflow Disruption: Build configurations can be time-consuming to set up and manage.

But all modern browsers (Chrome, Firefox, Safari, Edge) support HTTP/2 and "Import Maps" for JS modules. 

• Import maps enables us to use alias for full JS module paths - it is a method of creating a central registry that associates module names with their real locations is. This makes things clearer and more versatile by removing the requirement for you to write whole file paths in your code (across multiple files). 
• HTTP/2 is a more effective and rapid method of sending data across the internet. It makes it unnecessary to bundle files because browsers can handle several tiny files well. Instead of opening many connections to the server (like having many waiters running around), HTTP/2 uses one connection. Thus, multiple JS files can be downloaded concurrently, speeding up page load times.

Should we serve JS files from a CDN?

An excellent article that states why we need to actually use a CDN for serve popular JS libraries:  https://blog.wesleyac.com/posts/why-not-javascript-cdn

Excerpt from the article: 

"This means that one of these CDNs going down, or an attacker hacking one of them would have a huge impact all over society — we already see this category of problem with large swaths of the internet going down every time cloudflare or AWS has an outage."

Ruminating on Core Web Vitals

Have you ever clicked on a webpage only to spend a long time staring at a blank screen? Yes, it is frustrating. That's bad for the website (since you could just click away) and bad for you (because you're waiting).

This is where Core Web Vials are useful -- they are a set of metrics defined by Google to measure a website’s loading speed, interactivity and visual stability.  In essence, there are three things that websites must do well in order to ensure that users have a positive experience. 

• Quick loading (also known as Largest Contentful Paint, or LCP): This refers to how quickly a webpage's major content loads. A decent load time is defined as 2.5 seconds or less. 
• Smooth interactions (First Input Delay or FID): This is about how responsive a webpage feels. If you click a button and nothing happens for a while, that's a sign of bad FID. We want those clicks to feel instant, just like if you were pushing a real button. A decent speed is one that is less than 100 milliseconds.
• Stable visuals (Cumulative Layout Shift or CLS): This one's about how much the content on a webpage jumps around as it loads. Imagine you start reading a recipe and then all the ingredients suddenly jump to different places on the page - that's bad CLS! We want the content to stay put so you can focus on what you're reading. A score of under 1.0 is good.

More information on Core Web Vitals can be found here -- https://developers.google.com/search/docs/appearance/core-web-vitals

The top recommentations to improve the core web vitals are as follows:

• Optimize Image size and Image loading: Large, poorly optimized pictures are a primary cause of sluggish loading speeds.  Reducing the size of your pictures, using compression techniques, and turning on lazy loading—which loads images only when the user scrolls—will all help you get a higher Largest Contentful Paint (LCP) score. Also implement lazy loading - i.e. make images load only when visitors scroll down to them. 
• Caching:  Enable browser caching to store website elements on a user's device.  This way, the browser doesn't have to download everything all over again each time they visit your site. Leveraging a CDN would also help here. 
• JS & CSS optimization: Minify and compress your CSS and Javascript files.  This can significantly reduce their size and improve loading times.
• Preload: Preloading instructs the browser to fetch specific resources early, even before they're explicitly requested by the page. You can preload resources using the <link rel="preload"> tag in the <head> section of your HTML document.

Gaia-X & Catena-X: Data usage governance and Sovereignty

Unless you have been living under a rock, you must have heard about Gaia-X. The whole premise of Gaia-X was to build a fair and open data ecosystem for Europe, where everyone can share information securely and control who gets to use it. It's like a big marketplace for data, but with European values of privacy and control at its heart.

The core techical concepts that we need to understand around Gaia-X are "Data Spaces" and "Connectors".  

Data Spaces refer to secure and trusted environments where businesses, individuals, and public institutions can share and use data in a controlled and fair manner. They're like marketplaces for data, but with strict rules to ensure privacy, security, and data sovereignty (meaning individuals and companies retain control over their data).

A connector plays a crucial role in facilitating secure and controlled data exchange between different participants within the ecosystem. Think of it as a translator and bridge builder, helping diverse systems and providers communicate and share data seamlessly and safely. The Eclipse foundation has taken a lead on this and created the Eclipse Dataspace Component (EDC) initiative wherein many opensource projects have been created to build Gaia-X compliant connectors. 

These core concepts of Dataspaces and Connectors can also be used to build a modern data architecture in a federated decentralized manner. An excellent article on this approach is available on AWS here - https://aws.amazon.com/blogs/publicsector/enabling-data-sharing-through-data-spaces-aws/

An offshoot of Gaia-X is another initiative called Catena-X that aims to create a data ecosystem for the automotive industry. It aims to create a standardized way for car manufacturers, suppliers, dealers, software providers, etc. – to share information securely and efficiently through usage of standard data formats and procotols. The Eclipse Tractus-X™ project is the official open-source project in the Catena-X ecosystem under the umbrella of the Eclipse Foundation and has reference implementations of connectors to securely exchange data. 

But how do you ensure that the data is used only for the purpose that you allowed it to be used? Can you have legal rights and controls over how the data is used after you have shared it? This is the crux of the standards around Gaia-x/Catena X. 

At the heart lies the data usage contract, a legally binding agreement between data providers and consumers within the Catena-X ecosystem. This contract specifies the exact terms of data usage, including:

• Who can access the data?: Defined by roles and permissions within the contract.
• What data can be accessed?: Specific data sets or categories permitted.
• How the data can be used?: Allowed purposes and restrictions on analysis, processing, or sharing.
• Duration of access?: Timeframe for using the data.

Contracts establish a basic link between the policies and the data to be transferred; a transfer cannot occur without a contract. 

Because of the legal binding nature of this design, all users of data are required to abide by the usage policies just like they would with a handwritten contract. 

More details around data governance can be found in the official white paper --https://catena-x.net/fileadmin/_online_media_/231006_Whitepaper_DataSpaceGovernance.pdf

Besides contracts, every data access and usage event is logged on a distributed ledger, providing a transparent audit trail for accountability and dispute resolution. The connectors also enforce proper authentication/authorization through the Identity Provider and validate other policy rules. 

Thus Gaia-X/Catena-X enforce data usage policies through a combination of legal contracts, automated technical tools, independent verification, and a strong legal framework. This multi-layered approach ensures trust, transparency, and accountability within the data ecosystem, empowering data providers with control over their valuable information.