Deleting duplicate rows from a table

This seems to a favourite question during interviews - How to delete duplicate rows in a table?
Well, we can have many strategies:

- Capture one instance of the unique rows using a SELECT DISTINCT .., and dump the results into a temp table. Delete all of the rows from the original table and then insert the rows from the temp table back into the original table.

- If there is an identity column, the we can also write a query to get all the rows that have duplicate entries :
SELECT Field1, Field2, Count(ID)
FROM Foo1
GROUP BY Foo1.Field1, Foo1.Field2
HAVING Count(Foo1.ID) > 1
-----------------------------
Then loop thru the resultset and get a cursor for the query
"SELECT Field1, Field2, ID
FROM Foo1
WHERE Field1 = @FIELD1 and Field2 = @FIELD2".
Use the cursor to delete all the duplicate rows returned except one.

The following links discuss the above techniques and some more:

sql-server-performance
databasejournal
aspfaqs

Tools used in SDLC

I decided to list down all the available tools for different phases of a SDLC. Here it goes:

Requirements Management:
- DOORS: Dynamic Object Oriented Requirements System
- RequisitePro : A requirements management tool from Rational Software
- Requirements Traceability Management (RTM)

Design:
- ArgoUML: http://argouml.tigris.org/
-Rational Rose http://www.rationalrose.com/
-Enterprise Architect http://www.sparxsystems.com.au/
-Rhapsody http://www.ilogix.com/homepage.aspx
-ERwin http://www3.ca.com/solutions/Product.aspx?ID=260
-Rational Rose Real Time http://www-128.ibm.com/developerworks/rational/products/rose
-Describe http://www.embarcadero.com/products/describe/
-MagicDraw http://www.magicdraw.com/
-SmartDraw: http://www.smartdraw.com/examples/software-erd/
-Visio : http://www.mvps.org/visio/

Code Generation:
-OptimalJ: http://www.compuware.com/products/optimalj/
- Rational Application Developer
-Object Frontier
-JVider is Java Visual Interface Designer which helps with GUI building for Swing applications

Unit Testing:
-Jtest: http://www.parasoft.com/jsp/home.jsp
-JUnit: http://www.junit.org/

Load Testing:
SilkPerformer is load performance testing tool.
LoadRunner
JMeter
ANTS Load
Microsoft Web Application Stress Tool

Web Testing/ GUI testing:
SilkTest from Segue Software
Rational ROBOT
WinRunner

Configuration Management Tools:
Concurrent Version System - CVS
Subversion
Clearcase
Visual SourceSafe

Documentation:
Framemaker

Will keep augmenting this list...

Cool Free Data Models

While browsing the web, I came across this cool site which has over 200 data-models samples.
Great for anyone who wants to learn about data-modelling or have a template for a given problem domain.

The link is here : http://www.databaseanswers.org/data_models/index.htm

Some interesting data models I found in this site are - datamodels for Hierarchies , e-commerce etc. Worth a perusal.

Data Modelling Vs OO Modelling

While working on projects, often I had tussles with erst-while database developers who used to drive the design of the system using Data Models. Most of them came from non-OO background and were apt in ER diagrams. But the porblem is that a Data Model cannot and should not drive the OO model. I came across this excellent article by Scott Ambler discussing this dilemma.

http://www.agiledata.org/essays/drivingForces.html

In the above article, the author argues that object schemas can be quite different from physical data schemas. Using two lucid examples he shows how different object schemas can map to the same data schema and how it is possible for a single object schema to correctly map to several data schemas.
There are situations where OO varies significantly from ERD, for example in cases of inheritance and non-persistent classes.
Aslo, OO models can convey more specification on the class diagram in representing certain things like associations than ERD can.

I personally believe that if U are building a OO system, then we should first go for OO Class modelling and then think about resolving the OO-relational database impedance mismatch.

What are Multi-dimensional databases?

A multidimensional database (MDB) is a type of database that is optimized for data warehouse and online analytical processing (OLAP) applications. Multidimensional databases are frequently created using input from existing relational databases.

A multidimensional database - or a multidimensional database management system (MDDBMS) - implies the ability to rapidly process the data in the database so that answers can be generated quickly. A number of vendors provide products that use multidimensional databases. Approaches to how data is stored and the user interface vary.

Conceptually, a multidimensional database uses the idea of a data cube to represent the dimensions of data available to a user. For example, "sales" could be viewed in the dimensions of product model, geography, time, or some additional dimension. In this case, "sales" is known as the measure attribute of the data cube and the other dimensions are seen as feature attributes. Additionally, a database creator can define hierarchies and levels within a dimension (for example, state and city levels within a regional hierarchy).

More information can be at:
http://flame.cs.dal.ca/~panda/overview.html
http://www.dba-oracle.com/cou_mdb_class.htm

Ruminating on Database Cursors

Cursors are database objects that allow us to manipulate data in a set in a row-by-row basis or on a group of rows at one time. Cursors are quite popular among database developers as the row can be updated at the same time itself. There is no need to fire a separate SQL query.

But there is also a lot of confusion regarding the exact definition of cursors, because different people use it in different contexts. For e.g. when a database administrator talks of cursors, he means the server-side cursor he uses inside stored procedures. But if a JDBC application developer talks about a cursor, he may mean the pointer in the JDBC ResultSet object.
I did a bit of reseach on the web, and finally cleared a few cobwebs from the head. Here's a snapshot of what I learned:

• There are implicit cursors and explicit cursors. An implicit cursor is - well, that's just a piece of memory used by the database server to work with resulsets internally. Implicit cursors are not accessible via an externally exposed API, whereas explicit cursors are.
• Server-side cursors and Client-side cursors: The difference between a client-side cursor and a server-side cursor in classic ADO is substantial and can be confusing. A server-side cursor allows you to manipulate rows on the server through calls on the client, usually storing all or a portion of the data in TEMPDB. The client-side cursor fetches data into a COM object on the client. Its name comes from the fact that the buffered data on the client exhibits cursor-like behaviors you can scroll through and, potentially, update it. The behavior difference manifests itself in a few ways. Fetching a large resultset into a client cursor causes a big performance hit on the initial fetch, and server cursors result in increased memory requirements for SQL Server and require a dedicated connection all the time you're fetching. Client-side cursors can be dangerous because using them has the side effect of retrieving all records from the server that are a result of your command/SQL statement. If you execute a procedure or SQL statement that could retrieve 10,000 records, and use a client-side cursor, you had better have enough memory to hold 10,000 records on the client. Also, control will not return to your code/application until all of these records are retrieved from the server, which can make your application appear slow to users. If you suspect that you are going to retrieve a large number of rows, client-side cursors are not the way to go.
• DYNAMIC, STATIC, and KEYSET cursors: Dynamic cursors will show changes made on the base table as you scroll through the cursor. Static cursors copy the base table, to the tempdb. The cursor then reads from the tempdb, so any changes happening on the base table will not be reflected in the cursors scrolling. Keysets are in between Dynamic and Static cursors. A keyset will copy the base table's selected records keys into the tempdb, so the cursor will select the rows from the tempdb, but the data from the base table. So change to the base table will be seen, but new record inserts will not be.

Next, I tried to find out how these cursors can be manipulated using .NET and Java APIs.

In .NET, the DataReader object acts as a wrapper class for a server-side cursor. The DataReader provides a read-only and forward-only cursor. More info about this is here.

In JDBC, we handle everything using ResultSet objects. Since JDBC 2.0/3.0, we have ResultSets that are scrollable backwards and also that allow update operations. So I believe if U configure a ResultSet to be scrollable backwards and allow update opeartions, U are essentially dealing with a server-side cursor.

Code snippet:

Statement stmt = con.createStatement(ResultSet.TYPE_SCROLL_SENSITIVE,ResultSet.CONCUR_READ_ONLY);
ResultSet srs = stmt.executeQuery("SELECT COF_NAME, PRICE FROM COFFEES");
---------------------------------------------------------------------------------
The above code makes the ResultSet as scroll-sensitive, so it will reflect changes made to it while it is open. This is the equivalent of a dynamic cursor. In a dynamic cursor, the engine repeats the query each time the cursor is accessed so new members are added and existing members are removed as the database processes changes to the database.
The second argument is one of two ResultSet constants for specifying whether a result set is read-only or updatable: CONCUR_READ_ONLY and CONCUR_UPDATABLE.
Specifying the constant TYPE_FORWARD_ONLY creates a nonscrollable result set, that is, one in which the cursor moves only forward. If you do not specify any constants for the type and updatability of a ResultSet object, you will automatically get one that is TYPE_FORWARD_ONLY and CONCUR_READ_ONLY (as is the case when you are using only the JDBC 1.0 API). You might keep in mind, though, the fact that no matter what type of result set you specify, you are always limited by what your DBMS and driver actually provide.

Quick HEX dump utility in DOS

Recently, I wanted to see the hex dump of a binary file on a server. The server machine did not have my favourite hex GUI editors. I struggled to download a hex dump utility from the web, when my friend showed me how to use the 'debug.exe' tool in DOS to get a hex dump of a file.

On the DOS prompt type :
C:\>debug
-d
-----------------------
The 'd' (dump) option would dump the first 128 bytes. Typing 'd' again would give the hex dump of the next block of bytes.

U can type '?' to get a list of all the commands. Type 'q' to quit from debug mode.

Cool APIs for packet sniffing

A few years back, I often used to lament on the lack of packet sniffing API's available in Java or C#.NET. I knew of the WinPcap library, but I wanted a wrapper around it. Fortunately some good guys in the open source community have build these wrappers around WinPcap.

Check out the following links:
http://www.tamirgal.com/home/dev.aspx?Item=SharpPcap (A C# API)
http://sourceforge.net/projects/jpcap (A Java API)
http://www.winpcap.org/
http://www.tcpdump.org/

DataReader Vs DataSet in ADO.NET

Came across a wonderful article on MSDN which contrasts the use of DataReader Vs DataSet.
The article is available at : http://msdn.microsoft.com/msdnmag/issues/04/06/DataPoints/

Salient points of the article:

• A DataReader is a stream of data that is returned from a database query. When the query is executed, the first row is returned to the DataReader via the stream. The stream then remains connected to the database, poised to retrieve the next record. The DataReader reads one row at a time from the database and can only move forward, one record at a time. As the DataReader reads the rows from the database, the values of the columns in each row can be read and evaluated, but they cannot be edited.
• The DataReader can only get its data from a data source through a managed provider. The DataSet can also get its data from a data source via a managed provider, but the data source can also be loaded manually, even from an XML file on a hard drive.
• The DataReader supports access to multiple resultsets, one at a time, in the order they are retrieved. (Methods to checkout are NextResult() and Read() )
• When the SqlDataAdapter's Fill method is executed it opens its associated SqlConnection object (if not already open) and issues its associated SqlCommand object against the SqlConnection. Behind the scenes, a SqlDataReader is created implicitly and the rowset is retrieved one row at a time in succession and sent to the DataSet. Once all of the data is in the DataSet, the implicit SqlDataReader is destroyed and the SqlConnection is closed.
• The DataSet's disconnected nature allows it to be transformed into XML and sent over the wire via HTTP if appropriate. This makes it ideal as the return vehicle from business-tier objects and Web services. A DataReader cannot be serialized and thus cannot be passed between physical-tier boundaries where only string (XML) data can go.
• There are other times when a DataReader can be the right choice, such as when populating a list or retrieving 10,000 records for a business rule. When a huge amount of data must be retrieved to a business process, even on a middle tier, it can take a while to load a DataSet, pass the data to it on the business tier from the database, and then store it in memory. The footprint could be quite large and with numerous instances of it running (such as in a Web application where hundreds of users may be connected), scalability would become a problem. If this data is intended to be retrieved and then traversed for business rule processing, the DataReader could speed up the process as it retrieves one row at a time and does not require the memory resources that the DataSet requires.

User Defined Functions Vs Stored Procedures

Most databases have built-in functions, such as GetDate, DateAdd, and ObjectName. Such built-in functions are useful, but you can’t alter their functionality in any way, and that’s why UDFs are so powerful and necessary. UDFs allow you to add custom solutions for unique application-specific problems.
A UDF is actually a kind of subroutine that contains T-SQL statements and can return a scalar value or a table value. Hence U can call a UDF from a SELECT statement. Whereas a Stored Procedure needs to be invoked using the 'EXEC' command.
So when to use what? The answer depends on the problem situation. If you have an operation such as a query with a FROM clause that requires a rowset be drawn from a table or set of tables, then a function will be your appropriate choice. However, when you want to use that same rowset in your application the better choice would be a stored procedure.

The other differences btw a UDF and a Stored Procedure are as follows:

• A stored prodecure supports output parameters, whereas a UDF does not.
• A UDF can have a return value (scalar or table). A SPROC can create a table, but cannot return it. It can only return a integer (as status code)
• SPROCs are powerful enough to change server environment variables. UDFs are not.
• SPROCs can be used in the FOR XML clause but UDFs can't be.
• Whenever an error occurs in a UDF, the function stops immediately. In a SPROC, we must include error handling code, or the next statement is processed.
• U can 'join' to a UDF, but cannot 'join' to a SPROC.

Microsoft Enterprise Application blocks

Recently, Microsoft's Pattern & Practices Group has released the Enterprise Library, a large configurable and extensible software library that consists of seven integrated application blocks.

For more information and download visit http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnpag2/html/entlib.asp

The application blocks that comprise the Enterprise Library are the following:
Caching Application Block. This application block allows developers to incorporate a local cache in their applications.
Configuration Application Block. This application block allows applications to read and write configuration information.
Data Access Application Block. This application block allows developers to incorporate standard database functionality in their applications.
Cryptography Application Block. This application block allows developers to include encryption and hashing functionality in their applications.
Exception Handling Application Block. This application block allows developers and policy makers to create a consistent strategy for processing exceptions that occur throughout the architectural layers of enterprise applications.
Logging and Instrumentation Application Block. This application block allows developers to incorporate standard logging and instrumentation functionality in their applications.
Security Application Block. This application block allows developers to incorporate security functionality in their applications. Applications can use the application block in a variety of situations, such as authenticating and authorizing users against a database, retrieving role and profile information, and caching user profile information.

Ruminating on Transactions

Most of us have worked on transactions one time or the other. Transactions can be broadly classified as local or distributed.
Local transactions are of the simplest form in which the application peforms CRUD operations with a single datasource/database. The simplest form of relational database access involves only the application, a resource manager, and a resource adapter. The resource manager can be a relational database management system (RDBMS), such as Oracle or SQL Server. All of the actual database management is handled by this component.
The resource adapter is the component that is the communications channel, or request translator, between the "outside world," in this case the application, and the resource manager. In Java applications, this is a JDBC driver.

In a distributed transaction, the transaction accesses and updates data on two or more networked resources, and therefore must be coordinated among those resources.
These resources could consist of several different RDBMSs housed on a single sever, for example, Oracle, SQL Server, and Sybase; or they could include several instances of a single type of database residing on a number of different servers. In any case, a distributed transaction involves coordination among the various resource managers. This coordination is the function of the transaction manager. The transaction manager is responsible for making the final decision either to commit or rollback any distributed transaction. A commit decision should lead to a successful transaction; rollback leaves the data in the database unaltered

The first step of the distributed transaction process is for the application to send a request for the transaction to the transaction manager. Although the final commit/rollback decision treats the transaction as a single logical unit, there can be many transaction branches involved. A transaction branch is associated with a request to each resource manager involved in the distributed transaction. Requests to three different RDBMSs, therefore, require three transaction branches. Each transaction branch must be committed or rolled back by the local resource manager. The transaction manager controls the boundaries of the transaction and is responsible for the final decision as to whether or not the total transaction should commit or rollback. This decision is made in two phases, called the Two-Phase Commit Protocol.

In the first phase, the transaction manager polls all of the resource managers (RDBMSs) involved in the distributed transaction to see if each one is ready to commit. If a resource manager cannot commit, it responds negatively and rolls back its particular part of the transaction so that data is not altered.

In the second phase, the transaction manager determines if any of the resource managers have responded negatively, and, if so, rolls back the whole transaction. If there are no negative responses, the translation manager commits the whole transaction, and returns the results to the application.

Making web service proxy URL dynamic in .NET

In VS.NET, whenever we add a web-reference to a webservice, what happens behind the scenes is that a proxy is created which contains the URL to the webservice.

Now if we need to shift the webservice to a production server, do we have to create the proxy again?. (The IP address of the webservice server would have changed)

Fortunately there is a easy way out. Just right-click on the proxy in VS.NET and change its URL property from 'static' to 'dynamic'. And Voila !!!, VS.NET automatically adds code to the proxy class and web.config. The newly added code in the proxy will first check for a URL key in the web.config and use it to bind to the webservice.
So when we move the webservice to a different server, we just have to change the URL in the web.config file.

Object pooling in Java

Quite often, we may have to use object pooling to conserve resources and increase performance.
I came across some pretty good abstract components that can be used to develop a object pooling mechanism.

Check out the followling link to see all the classes that are available for use:
http://jakarta.apache.org/commons/pool/

An interesting aspect of the package was the clear separation of object creation from the way the objects are pooled. There is a PoolableObjectFactory interface that provides a generic interface for managing the lifecycle of a pooled instance.

By contract, when an ObjectPool delegates to a PoolableObjectFactory : -

makeObject is called whenever a new instance is needed.
activateObject is invoked on every instance before it is returned from the pool.
passivateObject is invoked on every instance when it is returned to the pool.
destroyObject is invoked on every instance when it is being "dropped" from the pool (whether due to the response from validateObject, or for reasons specific to the pool implementation.)
validateObject is invoked in an implementation-specific fashion to determine if an instance is still valid to be returned by the pool. It will only be invoked on an "activated" instance.

Doug Lea's Home Page

I have been impressed with the genius of Doug Lea when I saw his concurrent package in Java. His home page contains a lot of interesting links : http://gee.cs.oswego.edu/dl/

Chief among them are :

• A online OO book : http://gee.cs.oswego.edu/dl/oosdw3/index.html
• Roles Before Objects : http://gee.cs.oswego.edu/dl/rp/roles.html

This page is worth a perusal.

Thread pools in Java

I have often used thread pools in my applications. My favourite opensource thread pool was the PooledExecutor class written by Doug Lea (available here )

But it's also important to understand when to use thread pools and when not to use. I recently came across a good article on the web - http://www-128.ibm.com/developerworks/java/library/j-jtp0730.html

In this article the author argues that using thread pools makes sense whenever U have a large number of tasks that need to be processed and each task is short-lived. For e.g. Webservers, FTP servers etc.
But when the tasks are long running and few in number it may make sense to actually spawn a thread for each task. Another common threading model is to have a single background thread and task queue for tasks of a certain type. AWT and Swing use this model, in which there is a GUI event thread, and all work that causes changes in the user interface must execute in that thread.

Java 5.0 has come up with a new package java.util.concurrent that contains a lot of cool classes if U need to implement threading in Ur applications.

StringBuilder in Java 5.0

There is a new class in Java5.0 - StringBuilder class that can be used whenever we are doing some heavy-duty string manipulation and we do not wish to get bogged down by the immutable property of strings.
But then what happened to the old good StringBuffer class. Well, it's still there.
Here's what the JavaDoc says:

This StringBuilder class provides an API compatible with StringBuffer, but with no guarantee of synchronization. This class is designed for use as a drop-in replacement for StringBuffer in places where the string buffer was being used by a single thread (as is generally the case). Where possible, it is recommended that this class be used in preference to StringBuffer as it will be faster under most implementations.

Instances of StringBuilder are not safe for use by multiple threads. If such synchronization is required then it is recommended that StringBuffer be used.

Understanding Unicode

I still see many people with a lot of myths about Unicode. I guess the reason for this is that a lot of people still feel that Unicode is a encoding format that uses 16 bits to represent a character.
Let's put a few things in perspective here:

Unicode is a standard which has defined a character code for every character in most of the speaking languages in the world. Also it has defined a character code for items such as scientific, mathematical, and technical symbols, and even musical notation. These character codes are also known as code points.

Unicode characters may be encoded at any code point from U+0000 to U+10FFFF, i.e. Unicode reserves 1,114,112 (= 220 + 216) code points, and currently assigns characters to more than 96,000 of those code points. The first 256 codes precisely match those of ISO 8859-1, the most popular 8-bit character encoding in the "Western world"; as a result, the first 128 characters are also identical to ASCII.

The number of bits used to represent each code point may differ - e.g. 8 bits, 16 bits.
The size of the code unit used for expressing those code points may be 8 bits (for UTF-8), 16 bits (for UTF-16), or 32 bits (for UTF-32).
So what this means is that there are several formats for storing Unicode code points. When combined with the byte order of the hardware (BE or LE), they are known officially as "character encoding schemes." They are also known by their UTF acronyms, which stand for "Unicode Transformation Format"

UTF-8 is widely used because the first 128 bits in the byte are ASCII, and although up to four bytes can be used, only one byte is required for use in the English speaking world. UTF-16 and UTF-32 use a fixed number of bytes.

So to put in other words, Unicode text can be represented in more than one way, including UTF-8, UTF-16 and UTF-32. So, hey...what's this UTF ?

A Unicode transformation format (UTF) is an algorithmic mapping from every Unicode code point to a unique byte sequence. UTF-8 is most common on the web. UTF-16 is used by Java and Windows. UTF-32 is used by various Unix systems. The conversions between all of them are algorithmically based, fast and lossless. This makes it easy to support data input or output in multiple formats, while using a particular UTF for internal storage or processing.

For more information visit http://www.unicode.org/faq/

On Bytes and Chars...

During i8n and localization, we often come across basic fundamental issues such as:
- How many bytes make a character?
- How many characters/bytes are present in a string?

Each character gets encoded into bytes according to a specific charset. For e.g. ASCII uses 7 bit encoding, i.e. each char is represented by 7 bits. ANSI/Cp1521 uses 8-bit encoding, Unicode uses 16 bit encoding. UTF-8, which is a popular encoding set on the internet is a multibyte Unicode charset. So if someone asks - how many bytes make a character - the answer is - it depends on the charset used to encode the character.

Another interesting point in Java is the difference btw a 'char' and a character.
When we do "String.length()" in Java, we get the number of chars in the string. But a Unicode character may be made up of more than one 'char'.
This blog throws light on this concept: http://forum.java.sun.com/thread.jspa?threadID=671720

Snippet from the above blog:
---------------------------------
A char is not necessarily a complete character. Why? Supplementary characters exist in the Unicode charset. These are characters that have code points above the base set, and they have values greater than 0xFFFF. They extend all the way up to 0x10FFFF. That's a lot of characters. In Java, these supplementary characters are represented as surrogate pairs, pairs of char units that fall in a specific range. The leading or high surrogate value is in the 0xD800 through 0xDBFF range. The trailing or low surrogate value is in the 0xDC00 through 0xDFFF range. What kinds of characters are supplementary? You can find out more from the Unicode site itself.

So, if length won't tell me home many characters are in a String, what will? Fortunately, the J2SE 5.0 API has a new String method: codePointCount(int beginIndex, int endIndex). This method will tell you how many Unicode code points are between the two indices. The index values refer to code unit or char locations, so endIndex - beginIndex for the entire String is equivalent to the String's length. Anyway, here's how you might use the method:

int charLen = myString.length();
int characterLen = myString.codePointCount(0, charLen);

Preventing SQL injection attacks...

I still see a lot of applications that are vulnerable to SQL injection attacks bcoz of usage of dynamic SQL using string concatenation. It is a common myth that to circumvent SQL injection attacks we 'have' to use stored procedures.
Even if we are using dynamic SQL, it is pretty simple to avoid these attacks.

In .NET the following techniques can be used:
Step 1. Constrain input - Validate input using client side validation and server side validation (e.g. using regular expressions)
Step 2. Use parameters with stored procedures. There is one caveat with Stored procedures. If Ur stored procedure uses the 'EXEC' command which takes a string, then the same vulnerability exists there too.
Step 3. Use parameters with dynamic SQL. (Yepppiee...in .NET it's so simple to have named parameters even for dynamic SQL)

Code snippet :
-----------------------------------
SqlDataAdapter myDataAdapter = new SqlDataAdapter( "SELECT au_lname, au_fname FROM Authors WHERE au_id = @au_id", connection);
myCommand.SelectCommand.Parameters.Add("@au_id", SqlDbType.VarChar, 11); myCommand.SelectCommand.Parameters["@au_id"].Value = SSN.Text;
-----------------------------------

Other important points to be considred are using a least-privileged database account and avoiding disclosing error information to the user.
A good article discussing this is at: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnpag2/html/paght000002.asp

In Java, one can use 'Prepared Statements' and 'Stored Procedures' to prevent SQL injection attacks.

Common terms in Datawarehousing.

I have been interested in datawarehousing concepts for a long time, but unfortunately never got a chance to work on them. Here are some common concepts U need to know to understand any datawarehousing glibber.

Data Warehousing: An enterprise-wide implementation that replicates data from the same publication table on different servers/platforms to a single subscription table. This implementation effectively consolidates data from multiple sources.

Data Warehouse: A data warehouse is a subject oriented, integrated, non volatile, time variant collection of data. The data warehouse contains atomic level data and summarized data specifically structured for querying and reporting.

Data Mining: The process of finding hidden patterns and relationships in data. For instance, a consumer goods company may track 200 variables about each consumer. There are scores of possible relationships among the 200 variables. Data mining tools will identify the significant relationships.

OLAP (On-Line Analytical Processing)
Describes the systems used not for application delivery, but for analyzing the business, e.g., sales forecasting, market trends analysis, etc. These systems are also more conducive to heuristic reporting and often involves multidimensional data analysis capabilities.

OLTP (OnLine Transaction Processing)
Describes the activities and systems associated with a company's day-to-day operational processing and data (order entry, invoicing, general ledger, etc.).

Data Dictionary:
a software tool for recording the definition of data, the relationship of one category of data to another, the attributes and keys of groups of data, and so forth.

Reusable Application blocks for .NET

I often used to write some utility classes that would simplify the coding of most used tasks.
For e.g. accessing a DataReader for binding to a grid, we have to write all the database plumbing code again and again.
Hence MS has released some utility wrapper classes that can be used to simplify such routine tasks. A good article explaining this is at http://aspnet.4guysfromrolla.com/articles/070203-1.aspx

The application blocks can be downloaded from http://www.microsoft.com/downloads

How to implement Role authorization after Forms authentication

After authenticating a user using forms authentication, we may want to restict access to certain parts of the website to certain users - i.e. Authorize users.

To implement Role based authorization we would need to set up a database containing info about which role a user belongs to. Then we need to construct a Principal object specifying which role the user belongs to and assign it to the HttpContext user property.

An excellent article discussing this concept is at:
http://aspnet.4guysfromrolla.com/articles/082703-1.aspx

Advantages of using netmodules in .NET

I often wondered what was the advantage of compiling files into .net modules and linking them together using the Al.exe tool. I also noticed that al.exe actually only creates a stub dll, the netmodule files really have to be physically deployed too.

I guess the advantage of compiling to netmodules are:

• Each module may be independently developed by a separate set of developers.
• Each module may be written in any .net language.
• A multimodule assembly does have memory advantages: if you put rarely-usedtypes in one module, that module will only be loaded when one of its types is referenced, when the type is first used. If the rarely-used types are never used in an execution of the assembly, the module is not loaded.

GAC folder in Windows

When we go the default GAC folder in Windows located at : "C:\WINNT\assembly" we see the dlls that are registered in the GAC. But what if we have different versions of the same DLL registered in the GAC? How can a single Windows folder allow 2 files with the same name to exist?

To understand this, go to the DOS prompt and navigate to "C:\WINNT\assembly". Do a 'dir' command and see the contents. There would be a 'GAC' folder and inside the folder, there is a folder for each Assembly. Each assembly folder has a folder for each version. Inside this version folder there is the actual DLL. Try this out :)

XML databinding in Java using Castor

Often in my applications, I needed to persist some Java data objects as XML. For this I used to use JDOM which was quite cool to use bcoz of its Java-like API.
But there is one more XML data binding framework known as "Castor" which can handle all the marshalling and unmarshalling of object to XML for you automatically or with the help of a mapping file. Castor is indeed a cool XML data binding framework.

Furthur reading:
OnJava tutorial
http://www.castor.org/

XML parsing APIs in .NET

The .NET API has quite a few classes to help us manipulate XML data.
We have the 'XMLReader' class and its subclasses like 'XMLTextReader', 'XMLValidatingReader' ect that provide a forward-only, non-cached, read-only cursor to XML data. An interesting point is that XMLReader uses a 'pull model' unlike SAX push event model.

Then we have XPathDocument that provides a fast, read-only cache for XML document processing using XSLT. Quite good if U need to use XPath to query the XML data.

We have a special class for loading a DataSet directly into a XML form for manipultion. This class is the XMLDataDocument class that can load either relational data or XML data and manipulate that data using the W3C Document Object Model (DOM).

Diff btw Java and C#.NET

I came across this comprehensive document stating all the minute differences between Java and C#.NET. A good read :)

http://www.25hoursaday.com/CsharpVsJava.html

I still love Struts...

I have been a strong advocate of Struts right from the start. I often see developers getting confused over making a choice between struts, velocity, tiles etc.

The point is that "Struts" is a "controller-centric" MVC solution. The "View" and the "Model" parts are pluggable thanks to the plug-in architecture of Struts1.1
Hence is possible to use "Tiles" or Velocity templates inplace of plain JSP for the View in Struts.
Similarly the Model can be used to interact with EJB, JDO, Hibernate...etc.

The various choices available in Server-side Java may seem daunting to a new developer,but with experience we come to realize and appreciate the value of each solution and where it fits best :)

Diff btw 'package' access specifier in Java and 'internal' access specifier in .NET

At first I thought that the "package" access-specifier in Java is equivalent to the "internal" access-specifier in .NET. But actually there is a subtle difference.

The 'package' access-specifier in Java limits the access to a particular package/namespace, where as the 'internal' access-specifier in .NET limits the access to the containing assembly. Now a .NET assembly can contain more than one 'namespace' or package.
So is there any way to restrict the scope of a member of a class to the same namespace?

Suprisingly NO, there is none in .NET..So if we were designing by 'separation of concerns', maybe it would make sense to map one namespace to one physical assembly.